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analyzing  difficult  decision  problems.  The  results  of  the 
experiment  show  that  OPINT  aids  in  the  decision  making  process 
but  has  some  severe  limitations  as  it  currently  exists. 
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ABSTRACT 


A  C  ^  i  n  c  t  j  d  e  s  the  nat  j  Tatherna  (nr 
intell  i  3<“nc°)  orocss5»  the  decision  making  process/  and  the 
operating  forces  reouired  to  achieve  a  particular  mission. 
Unfortunately/  the  majority  of  the  resources  spent  in  the 
improvement  of  these  systems  are  expended  in  either 
oroviainn  natter  data  gathering,  failsafe  communications/  or 
better  weapon  systems.  Little  thought  is  given  to  how  the 
decision  processes  the  data  in  oroer  to  make  effective 
decisions.  Several  computerized  decision  aids  have  been 
suaoested  to  fill  this  can.  One  of  these  aids/  Operations 
and  Intelligence  ( 0 P I \ T )  is  evaluated  to  assess  its  utility 
as  a  computerized  decision  aid  in  C3  applications.  ORIuT  is 
an  on-line/  interactive/  real-time  decision  aia  which 
assists  decision  makers  by  prescribing  a  straightforward 


normative  procedure  for  oroaoizi 

ng 

and 

analyzina 

d  i  f  f  i 

cult 

decision  problems.  The  results 

0  f 

the 

exrpr 

i  men  t 

Show 

t"at 

OP  I  mi  t  aids  in  the  decision  m  a  k  i 

ng 

process  / 

but 

has 

some 

severe  limitations  as  it  currently  exists. 


a 


LIST  OF  FIGURES 


1.  ACT  A :'i 0  EvENT  F OR  *\ S - 

2.  OEC  [ S I Q'i  TREE  STRUCTURE - 

3.  DECISION  TREE  .\IT.h  VALUES  ADDED - 

U.  DECISION  TREE  A  I T  H  PROBABILITIES  ADDED-- 

5.  DECISION  TREE  wITh  EXPECTED  VALUES - 

6.  C  3  SYSTEM - - - . - . . 

7.  HIERARCHICAL  DATA  STRUCTURE - 

«.  PROTOTYPE  SPATIAL  DATA  MANAGEMENT  SYSTEM 
<5.  DATA  ACCESS  CONTROLS - 

10.  ADAPTIVE  INFORMATION  SELECTION  SYSTEM  CONCEPT 

1 1 .  ULTRA-RAPID  READING - 

12.  CRISIS  MANAGEMENT  DECISION  AID - 

13.  CRISIS  MANAGEMENT  PROBLEM  ANALYZER . . 

1A.  CRISIS  DESCRIPTOR - 

15.  THE  GROUP  DECISION  AID  PROCESS - 

1b.  DECISION  mqoeL  format - 

17.  PEGRET  MATRICES - 

16.  COMBINED  VALUE  REGRET  MATRIX - - - 

IP.  EXPECTED  VALUE  MATRIX - 

20.  THRESHOLD  MATRIX - 

21.  MANUALLY  CHANGED  EVENT  PROBABILITIES - 

22.  EXPERIMENT  SCHEDULE - 


T  .  P  E  3  C  p  T  P  T  I  f  i M  OF  DECISION  ANALYSIS 

Durir.a  tites  of  crisis  or  war,  there  is  orobabl  y  no 
field  of  endeavor  like  military  and  strategic  command  ana 
control  where  Decisions  are  made  ou i c  k 1 y  ,  but  under 
ccnaitions  of  risk  and  extreme  uncertainty.  This  is 
especially  true  since  accurate  infor-at ion  on  all  aspects  of 
a  rroolem  are  almost  always  concealed  from  decision  makers. 
Every  military  force  tries  to  conceal  itself  and  makes  every 
at  tenet  to  deceive  the  opposing  decision  ma<er  as  to  its 
intentions.  Facts  surroundina  the  tactical  situation  are 
elusive?  escecial ly  those  concerning  the  eneny.  These  tynes 
of  decisions/  then,  are  difficult  and  f renuent I y  would  seem 
to  defy  a  systematic  decision  makino  process  (Hecartment  of 
the  Army,  1930,  o .  3-2).  At  the  same  time,  there  is  no 
arena  in  which  bad  decisions  have  more  traq'C  results.  The 
cost  of  makino  errors  can  crow  exponential  ly  mainly  due  to 
the  comolex  r e 1  a t i on s n i o s  and  resulting  cnain  reactions,  or 
the  errors  could  cancel  themselves  out  and  the  costs  not  be 
immediately  known  (Turban  and  Meredith,  1977,  p.  4). 

Con seuuen t 1 y ,  while  decision  makers  must  recognize  that 
orderly,  rational  decision  makino  orocedures  are  difficult, 
trey  must  at  the  same  time  constantly  strive  to  base 
decisions  on  the  most  rational  thouaht  orocess  oossiMe.  To 
reduce  error  probability,  ana  for,  the  sake  of  survival, 
decision  makers  must  become  more  sophisticated  (Turban  and 
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v  e  r  e  d  i  t  h  ,  19  7  7,  p .  51.  They  must  learn  to  utilize  new 
tools  and  techniques  that  are  b  e  i  n  a  developed.  No  one  could 
imagine  a  successful  suroeon  utilizing  equipment  ano 
procedures  from  toe  turn  of  tne  century.  Yet/  in  decision 
making*  you  can  still  find  decision  makers  usinq  the  tools 
and  techniques  of  that  tine. 

Military  anq  strategic  decisions  are  not  only  made 
under  conditions  of  uncertainty  (where  not  all  facts  are 
available)  and  stress*  they  are  also  rarely  considered 
final.  The  ever  changing  situation  brings  with  it  the 
requirement  to  continuously  revise  appraisals*  estimates* 
and  perhaos  decisions.  This  occurs  since  decision  makino  is 
based  on  the  future  but  is  dependent  on  the  oast.  For 
years*  managers  have  considered  decision  making  to  he  a  Dure 
art  or  talent  which  is  acquired  over  a  long  period  of  time 
through  experience  or  trial  and  error.  It  has  been 
considered  an  art  because  a  wide  variety  of  individual 
styles  can  oe  used  in  approaching  and  successfully  solving 
t  n  same  tvoe  of  nrnnl  pts.  One  woul~  normally  base  these 
styles  on  creativity*  judgment*  intuition*  and  experience 
rather  than  some  sort  of  systematic  method  (Turban  anq 
Mereditn,  1  ^  7  7  r  p.  5). 

Decision  making  as  a  discipline  had  its  origins  in 
operational  analysis  techniques  beoinnina  in  world  if  a  r  II 
(williams*  lQ7h,  o.  12).  These  techniques  were  tvoically 
applied  to  special  tvoes  of  clear-cut*  recetitive  orob'ems* 
such  as  those  of  systematic  search  and  resource  allocation. 


Since  tup  1  Q  6  0  '  s  ,  ho  w  e  v  e  r  ,  a  po  re  general  technology 
has  emerge ct  tnr  imposing  1  o  a  i  c  a  1  structure  cn  the  reasoning 
tuat  unnerl ies  any  specific  decision.  Inis  technology  is 
decision  analysis  (“arc  1  ay  et  a  1  .  »  19/7,  c .  i v)  .  Since 
1°7G,  tnere  has  been  a  major  effort  by  defense  agencies  to 
adoot  this  technology  to  their  day-to-day  decision  making. 
N'anv  have  found  it  a  wav  to  make  better,  more  defensinle 
decisions. 

Decision  analysis  is  a  quantitative  method  «h i cn 
oermits  the  systematic  evaluation  of  the  costs  or  benefits 
accruim  from  courses  of  action  that  might  be  taken  in  a 
decision  oroblem  (Barclay  et  a!.,  1977,  o.  vi).  The  method 
includes  the  i oen t i f i c a t i on  of  the  alternative  choices 
involved,  the  assignment  of  values  (costs/henef i ts)  for 
oossicle  outcomes,  and  the  exoression  of  the  probability  of 
those  outcomes  occurring.  Once  this  is  done,  the  orobable 
aain  or  loss  associated  with  each  alternative  can  be 
determined  bv  systematically  combining  the  probabilities  and 
va 1 ues . 

Ip  addition  to  the  primary  role  of  decision  analysis  as 
a  method  for  the  logical  solution  of  complex  decision 
problems,  it  also  has  several  additional  advantages  as  well. 
The  formal  structure  of  decision  analysis  insures  all  the 
o I pmpnts,  their  relationshios,  and  their  associated  weights 
have  b©en  considered  in  the  decision  problem.  The  model  of 
the  decision  problem  can  serve  an  important  role  in 
facilitating  communications  between  those  involved  in  the 


decision  process.  Also  if  is  very  easy  to  identify  the 
areas  of  disaoreement,  their  relative  importance*  and  if 
t~ey  actually  h  a  v  e  anv  m  a  t  a  r  i  a  1  impact  on  t  ^  e  indicated 
decision.  final  iv,  wren  chanoes  occur  in  the  problem,  it  is 
relatively  easy  1 3  reenfpr  the  existino  problem  structure  to 
chanoe  values  or  to  add  or  remove  problem  dimensions  as 
reouirea  (Barclay  et  al.,  Id77,  d.  via). 

It  should  be  erohasi 7Pd  that  in  no  sense  does  decision 
analysis  replace  decision  makers  or  toe  role  of  human 
iudoment  in  decision  •■skina.  Intuitive,  implicit*  or 
iuoomental  decision  m  a  '<  i  n  a  is*  after  all*  the  mainstay  of 
tne  experienced  decision  maker  and  for  qooo  reason. 
Intuitive  un- aided  methods  have  typically  served  the 
decision  m  a  <  e  r  well*  ano  n  a  /  s  h  e  can  reflect  on  some  personal 
history  of  Productive  reliance  on  their  develooino  intuition 
(tirirnprg*  1  3  7  <? ,  o.  1  °  1  1  .  Tn?r»  is  considerable  evidence 
tc  show  that  unaided  decision  marine  is  reasonable* 
effective,  and  rolianle  (°eterson  and  Beach*  1  3  5  7  ,  dp. 
do-ay,).  ‘-‘ovever,  what  decision  analysis  does*  is  provide  an 
orderly  ano  Tore  easily  understood  structure  that  helos  to 
a-doreoate  the  wisdom  of  exoerts  on  the  many  tooics  that  may 
he  needed  to  make  a  decision*  and  then  suooert  the  skilled 
decision  maker  by  providing  him  with  sound  techniques  to 
supplement  and  ensure  the  internal  consistency  of  his 
judgment. 

Complex  decision  problems  are  often  difficult  fo 
resolve.  This  occurs  f^r  a  number  of  reasons.  Options  are 


not-  always  c  I  a  a  r  I  v 


oe<  i  ned 


Any  results  which  may  be 


derived  from  the  selection  of  a  (Particular  option  may  dp 
hi  only  uncertain.  Also,  it  is  often  difficult  to  determine 
relative  preferences  for  the  onssinle  decision  outcomes, 
■.•hen  problems  such  as  these  do  occur,  the  decision  ma<er 
normally  takes  steps  to  structure  the  problem  and  reduce  it 
to  a  more  explicit  form.  This  is  exactly  what  decision 
analysis  does. 

decision  analysis  nuilos  uoon  four  basic  elements  which 
are  inherent  in  any  decision  problem  (tsrc 1 av  et  al.,  1977, 
o.  1).  The  use  of  these  four  elements  allows  for  a  smooth 
procedure  in  the  resolution  of  complex  decisions.  The  four 
elements  are: 

1.  A  set  of  initial  courses  of  action.  You  must 
have  more  than  one  alternative  cr  there  is  no 
decision  to  be  made.  All  possible  alternatives 
snould  be  considered  without  regard  to 
rlausibilitv  a*-  this  point. 

?.  The  possible  conseauences  of  each  initial  act. 
These  must  be  considered.  What  are  the 
important  things  that  can  happen  that  will  make 
one  act  more  valuable  or  worth  more  than  another 
act?  Relevant  seauences  of  subseauent  events 
ano  follow-uo  acts  must  be  identified  for  each 
initial  act. 
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These  four  elements/  as  described/  orovide  a  way  to 
organize,  ouantifv,  and  trace  the  loaical  implications  of 
roe  decision.  The  nri-rarv  objective  is  to  provide  a  model 
cf  a1-  least  part  (all  would  be  best)  of  the  decision.  The 
use  c f  the  worn  model  in  this  case  means  to  represent  the 
decision  in  a  quantifiable  form. 

It  is  a  central  orecent  of  decision  analysis  that  all 
roievan*-  considerations  in  a  Decision  can  be  represented 
fully  in  a  decision  o  i  =  o  r  a  m  (Barclay  et  a  I  .  ,  1977,  o.  2). 

This  decision  diagram  will  show  everyfning  a  decision  maker 
fee's  is  relevant  to  the  Problem  in  Question.  A  decision 
diagram  consists  essentially  of  a  network  of  branches 
cor resoondi ng  to  possible  seouences  of  acts  and  events, 
farm  i no  cut  from  an  orioi n  at  the  left  to  a  time  horizon  at 
the  right.  Acts  are  available  choices.  Events  are  possible 
occurrences  which  are  oart'y  or  comoletelv  outside  the 
decision  maker's  control,  though  the  chance  of  onp  of  them 
may  be  influenced  by  acts  which  were  carried  out 

1? 


h  a  n  o  e  n  i  n 


earl  j  e r 


Toe  iec  i  3  i  on  i  i  a  q  r  e  m  or^rhicil  tv  Ti  st  i  nqui  shes  acts 
f  ron  events.  Jc  t  f o r <  s  are  reoresenten  hy  squares  and  event 
forks  are  represented  try  circles  as  shown  in  F  i  cure  1. 

These  act  fort  s  and  event  forks  are  then  comoineT  to 
f o r ti  a  decision  tree.  An  e  x  a  m  o  I  e  of  a  tree  is  shown  in 
F i oure  2.  Tnis  decision  situation  involves  a  company  which 
must  dec i do  whether  to  rid  on  two  projects*  A  and  H  .  The 
decision  on  project  A  must  he  made  orior  to  the  decision  on 
-?  . 

Tnis  decision  diaqram  depicts  all  the  oossicle  acts  and 
events  and  shows  how  these  relate  to  each  other  in  the 
decision  situation.  thile  the  reoresentation  would  help  a 
decision  maker  to  see  at  a  q  1  a  n  c  e  his  alternatives  and 
identify  those  trims  that  m  i  a  h  t  affect  any  choice  to  be 
made*  it  does  not  yet  answer  the  central  ouestion:  which 
choice  should  he  male?  That  question  can  net  be  answered 
withou*-  considering  the  value  of  the  oossicle  outcomes  and 
the  liyelihqod  of  occurrence  of  the  events.  In  order  to 
show  how  this  is  incorporated  into  the  decision  model*  let's 
return  to  the  ni Hi  no  example.  Let’s  say  it  has  been 
calculated  it  will  cost  310*00  to  prepare  a  bio  for  either 
project.  Tf  the  hid  on  A  is  won*  a  oain  of  350*000  will  be 
realized.  Project  *5  will  also  return  a  oain  of  550  *  000  if 
the  hid  is  won  hut  due  to  overhead*  will  onl  v  return  a  qain 
of  3 2 n , 0 0 0  if  bid  A  is  also  won.  The  decision  tree  now 
looks  like  Fioure  3.  The  numbers  shown  at  the  right  side  of 
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IGURE  3:  DECISION  TREE  WITH  VALUES  ADDED 
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the  diagram  are  the  oath  values.  These  values  are  the  sum 
of  the  costs  and  q  a  i  n  s  atom  each  n  a  t  h  . 

t.  cm  to  coTolet®  the  diagram,  we  nee  a  to  o  o  r  t  r  a  y  the 
degree  of  uncertainty  about  tne  events  we  cannot  control# 
soecifically  wnetner  a  oid  is  won  or  lost.  Based  uoon  orior 
exoerience#  it  is  jugged  that  the  bid  on  oroject  A  is 
equally  likely  to  be  won  or  lost  (Drobability  .5  win,  .5 
lose).  If  A  is  won,  rne  oronability  of  winning  the  bid  on  ri 
is  reduced  to  .  d  .  '■'owever,  if  is  lost,  the  chances  of 
winning  B  increase  to  .7,  and  if  A  is  not  bid  on,  the 
chances  for  B  are  .6'  to  win.  Aodinc  these  values  leaves  our 
diagram  looking  like  Pioure  d . 

This  diagram  should  now  be  a  virtually  comolete 
translation  or  -nooel  of  the  cerceof  i  on  of  the  decision  maker 
of  toe  decision  rrohl e".  All  that  is  left  now  is  to 
determine  which  is  fbe  best  solution.  The  method  to  do  this 
involves  tne  calculation  of  weighted  values,  often  called 
exceeded  v  a • ue  s ,  for  each  decision  ootion.  This  technique 
is  tne  simplest  and  statistically  most  straightforward 
method  that  can  oe  used.  Care  must  be  taken  however,  since 
frequently  there  are  decision  circumstances  wherein  an 
exoected  value  solution  to  a  decision  oroblem  may  not  be  an 
optimal  one.  In  these  cases,  alternative  means  of  treat i na 
value  are  renuireo  and  will  be  discussed  later. 

Determining  a  value  for  each  act  in  a  decision  is  done 
by  a  crocedure  called  folding  back  the  decision  diagram. 


Values  ar*>  substituted  for  each  act  and  event  fork  beginning 
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FIGURE  4: 


DECISION  TREE  WITH  PROBABILITIES  ADDED 


1  * 


+  30K 


+  40K 
+  3  OK 


-  2  OK 

-10K 

+  40K 


-10K 

0 


at  the  ri-jhr-hflni  side  of  the  a i aor am .  At  eacH  event  fork* 
tne  exoact  ei  value  of  that  event  is  calculated  by 
m  u  1  t  i  o  1  v  i  n  o  the  value  of  each  c  o  s  s  i  b  1  e  o  u  f  c  o  m  e  by  the 
orotiahi  I  i  tv  of  the  occurrence  of  that  outcome.  At  each  act 
font*  the  act  witn  the  hiqnest  value  is  the  one  which  should 
aive  the  oest  decision.  The  crocess  of  substitution  is 
continued  until  the  initial  act  fork  is  reached.  Figure  5 
depicts  this  orocedure  •  Therefore  the  decision  ma<er  would 
bid  on  oroject  A  ,  then  if  he  <*  o  n  tne  bio,  not  bid  on  Project 
R  .  If  he  lost  the  gig  on  A,  he  would  oid  on  project  d . 

Thus  far  in  the  discussion  of  decision  analysis*  the 
values  of  toe  gains  or  losses  were  assigned  in  terns  of  only 
one  measure*  money.  The  outcomes  were  then  compared*  using 
the  expected  v  a  1  u  °  *  very  easilv  since  it  is  easy  for  us  to 
relate  to  money  as  a  value.  However*  in  most  real  life 
situations,  especially  those  of  the  military*  aains  and 
losses  are  not  "“osure'i  in  money*  rather  in  terrain* 
enuiomenr,  capabilities*  and  even  lives  (Williams*  1978*  p. 
<?<?).  These  value  dimensions  are  Qualitative  rather  than 
guantitative  and  are  very  difficult  to  measure.  To  further 
cloud  the  issue*  each  individual  has  his  own  personal 
subjective  values  for  each  of  these. 

In  addition*  very  seldom  are  these  value  dimensions  or 
attributes  given  e  o  u  a  1  weioht  in  a  decision.  For  example* 
consider  the  selection  of  a  radio  set  from  several  comoetina 
models.  Cost  mgv  ne  an  important  consideration*  but  so  are 
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EV(7)  «  ( .4) (50K) + ( . 6) (30K)  =  38K 
EV{8)  =  ( . 7) (30K) + ( . 3) (-20K)  =  15K 
EV(6)  =  (.6) (40K)+(.4) (-10K)  =  20K 
EV(4)  *  LARGER  OF  38K  OR  40K  =  40K 

EV{5)  «  LARGER  OF  15K  OR  -20K  =  15K 

EV(3)  *  LARGER  OF  20K  OR  0  =  20K 

EV(2)  =  ( . 5 )  ( 4  OK)  +  ( . 5)  (15K)  =  27. 5K 

EV(1)  *  LARGER  OF  20K  OR  27. 5K  =  27. 5K 


FIGURE  5: 


DECISION  TREE  WITH  EXPECTED  VALUES 


important  each  is  r  e  o  u  i  r  e  s  neter'i'ination,  /"hat  is  important 
in  tl'a  decision  is  which  of  these  at  tritxites  is  more 
important  r  n  a  n  c  n  e  others  and  how  is  this  difference  scaled. 
Then  now  ho  tne  r  e  m  a  i  n  i  n  a  attricutes  fit  on  this  scale.  For 
instance*  are  cost  and  reliability  eaually  important?  If 
not/  what  exactly  is  the  difference  and  how  does  the 
decision  maker  make  trad  e-offs  between  the  two. 
determinations  of  this  sort  must  be  made  for  all  attrioutes 
and  somehow  o  u  h  mo  a el  must  oe  ar 1 e  to  comcare  across  these 
many  attrioutes  and  aonreoate  the  results  to  indicate  the 
best  solution. 

Problems  of  fhis  nature  have  sourred  the  development  of 
multi-attrioute  utility  models  (williams/  1978/  o.  24). 
These  models  help  to  provide  a  relative  ranking  for 
attributes  and  also  provide  a  common  uuide  for  aaqreaating 
the  measures  into  a  sinole  index  of  worth.  This  process 
involves  toe  assianment  of  a  measure  of  utility  or  merit  to 
the  attributes. 

Utility  can  be  described  as  a  subjective  measure  of 
"liking"  (Barclay  et  a  1  .  /  1977,  o.  27).  It  is  a  Personal 
value  reflecting  how  you  subjectively  value  something.  The 
application  of  utility  as  a  measure  for  the  value  of  an 
attrioute  or  alternative  was  first  proposed  by  Von  Meuman 
and  Mo r gens t e m  .  T rev  suaaested  that  each  individual  has  a 
measurable  preference  among  various  choices  available.  The 
preference  they  called  utility  and  is  measured  in  arbitrary 
units  called  ufi'es  (Von  Meuman  and  ‘’'ornpnstern,  1°44,  o. 


27).  utility  is  base'i  on  the  concent  that  in  decision 
•nayirii  a  reason  will  choose  that  alternative  which 
01  a  x  i  m  i  z  e  s  o  i  s  nr  mar  axoected  utility  • 

,;'j1  t  i  -at  t  r  tout  e  utility  models  have  been  used 
extensively  in  tte  systems  acquisition  role  ( w  i  1  1  i  a  it  s  ,  1979, 
o.  <2  S )  .  Tne  oroceoures  encouraae  discussion  amonq  the 
decision  »a<er  and  his  staff*  do  longer  can  cost  be  a 
little  mo  re  important  than  reliar,  ility.  It  is  now,  neroaos, 
ten  utiles  more  imoortant.  r nee  tne  values  are  ass i ane  i  to 
tno  model,  a  no  a  rest  decision  decided  uDon,  the  mode)  no« 
allows  you  to  vary  t*e  relative  weiohts  to  see  which  ones 
have  the  most  i~oact  on  the  cronosed  solution.  This  is 
called  a  sensitivity  analysis  ano  halos  decision  makers  when 
mey  are  uncertain  about  the  accuracy  of  their  information. 

Decision  analysis  arolies  dayesian  techniques  to  ol ace 
a  value  on  information  that  reduces  uncertainty  (been  and 
Scott  ’-’erton,  117h,  o.  uu).  Bayesian  techniques  oroviti? 
formal  mothodolooies  for  anal yzinq  the  imolicat'ons  of  a 
decision  maker's  suojective  judgment  of  probabilities, 
uodafinq  these  assessments  as  additional  information  is 
obtained.  This  additional  information  can  result  in  one  of 
two  events:  either  the  uncertainty  is  comoletelv  removed  or 
the  additional  informal  ion  allows  the  decision  maker  to 
revise  the  initial  (orior)  assessment  of  the  d r oh ah  i  1 i t i e s 
(keen  and  Scott  v or  r  on ,  1 °7  % ,  n.  40). 

is  stated  earlier,  th#  use  of  decision  analysis  does 
net  reolace  the  decision  maker  nut  ados  a  new  dimension  to 
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1 


his  fipcision  maviog  caoanilities.  It  is  also  assumed  the 
use  of  tnese  manual  techniaues  could  slow  the  decision 
m  a  <  i  m  orooress  cons  i  neracl y<  especially  if  a  fear  of 
»3thon’stics  exists.  Po  r  c  una  c  e  1  y  ,  »hat  ias  tieen  developed 
are  decision  ails  which  Greatly  ease  the  complexity 
involved. 

A  decision  aid  is  a  human-system  interface  designed  for 
toe  soecific  ourocse  nf  sucoon inn  and  enhancing  a  decision 
-raver  in  “>  i  s  decision  mavjna  role  (Keen  ana  Scott  V  orton, 

1  ^  7  h ,  n .  So).  It  is  a  ton!  for  use  by  the  decision  maker. 
Decision  aids  ar<=  normally  stored  on  comnuters  as”  the  use  of 
fne  computer  reduces  all  needs  for  calculations  oy  the 
decision  m?i<er  and  heirs  to  speed  the  decision  crocess. 

The  general  availarilitv  of  low  cost*  hiah  capacity* 
fasf  i n f o r ma t i on  crccessinc  rechnoloay  hss  enabled  man  to 
extend  and  increase  his  intellectual  capacity.  The  effect 
of  this  has  be°n  that  man  can  now  deal  mere  effectively  witn 
complex  matters  on  his  own  complex  ter»?  as  aoainst  the  more 
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ore-informat  ion 

t  ec  hno 1 oay 

rt)an 

(OrinVprs 

*  1972*  v . 

5). 

The  focus 

is  not 
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for 
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information  that  is  available  and  the  scientific  methods 
which  are  in  development  and  use. 

The  use  of  decision  aids  normally  places  increased 
demands  unon  human  creativity  and  judoment  rather  than 
re'ievino  the  dec i S i nn  r a<er  of  the  need  to  exercise  them  as 
midht  be  expected.  Tnis  is  because  the  availability  and  use 
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0^  decision  a  i 03  which  serve  as  an  extension  of  tee  decision 


m  a  k  e  r  '  s  intellect  rrovioe  new  and  interesting  oocortuntties 
for  usioo  nis  intellect.  decision  aids  helo  the  decision 
maker  ov  orescrinino  a  straightforward  normative  crocedure 
for  croanj^in.g  and  analyzing  difficult  decision  o  r  o  o  1  e  m  s 
involving  0  o  t  r  uncertainty  stout  the  outcome  of  future 
events  and  oerolexity  al-'out  the  complex  value  tradeoffs 
involve, g  in  th»  choice  of  a  course  of  action  (drinkers* 
r.  7). 


! 

1 


T  T  .  M5F  PF  r.'frf.  ISI» ) \  MhS  7  T  ^  1 1*1  TnF  C3  &c t N a 


C  o  m  m  a  n  d  ,  co"M”o  I  and  communications  (  C  3  )  can  d  e  defined 
as  a  orocess  which  o  r  o  v  i  d  9  s  the  commanoer  (on  decision 
maker)  with  a  means  of  receiving  information,  making 
decisions  h  a  s  e  o  on  this  information,  and  then  incrementing 
and  monitoring  the  oecisicn  in  order  to  achieve  Ms  or  her 
mission  f  ■'  o  o  s  e  ,  \  ^  r  o  i  .  \  C  3  system  includes  the  data 

gathering  (or  intelligence)  orocess,  the  decision  making 
rr oc®ss,  and  tKe  ocerating  forces  (or  weaoons)  reouired  to 
achieve  the  mission.  4  C3  system  would  1  o  o  «  somewhat  like 
the  f  i  aure  Shown  r.  slow. 


FIGURE  6:  C3  SYSTEM 
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This  figure  snows  that  data  is  gathereg,  ooss  i  d  I  / 
tnrou'ih  tHo  use  o*  early  warn  i  no  radarsi  ohotoaraohy,  or 
reports  (ro»  e^rin,  am  sent  to  a  con-nana  renter.  Once 
wi f  K i n  tne  command  center,  the  g  a  t  a  is  analyzed  any  a  de¬ 
cision  is  made  recaraioo  the  use  of  weapons  or  units.  These 
decisions  are  then  transmitted  to  the  respective  units  tor 
jntc'lenentation.  Requirements  tor  data  or  information  could 
originate  a*’  the  command  center  or  it  could  be  spontane¬ 
ously  generated  as  in  the  case  of  a  missile  attac<. 

unfortunately,  the  majority  fit  net  all)  of  the 
resources  soenf  in  the  improvement  of  these  systems  are 
expended  in  either  providing  cat  far  data  aatherina  (suen  as 
neater  radars),  failsafe  communications,  or  ciaoer  and 
better  wearers  systems.  These  are  all  important  aspects  and 
should  oe  improved,  but  this  is  also  indicative  of  the  lack 
of  appreciation  tor  what  must  oe  done  with  f  h  e  data  when  it 
is  delivered  to  f  t>e  command  center.  Little  thouatt  is  given 
to  now  the  decision  maker  orocesses  the  the  data-in  order  to 
make  effective  decisions  and  forecasts. 

The  amount  of  data  available  today  to  decision  makers 
is  m  i  ndbodd  1  i  np  .  PoaM  st  ical  1  v,  if  methods  for  cooni  t  i  ve 
information  orocessina  are  not  improved  at  the  same  rate  as 
our  communications  can  provide  it,  much,  if  not  all  of  our 
sophisticated  communications  tecfnoloav  will  go  under 
utilized.  what  is  needed  is  more  research  aimed  at 
improving  our  information  mananement,  decision  making,  and 
forecasting  utilizing  *•  h ®  applications  available  through  the 
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Decision  Ti^k  i  nn  tias  at  f  h«  heart  of  the  C  3  crocess 
(  An d  r  i  o  1  9  ,  1  ^  >  ,  p.  ig).  Tne  entire  Cs  system  focuses  on 

o  r  o  v  i  o  i  n  a  a  -earns  to  transmit  ail  r  e  o  u  i  r  a  o  aata  to  the 
decision  maker  eoo  then  a  means  to  transmit  the  aecision  to 
the  required  oarties.  Therefore#  C3  svstems  which  do  not 
contain  some  sort  of  decision  makinq  suDOort  should  be 
chjl l»nieo  because  they  ere  incomplete. 

*‘cst  decision  makers  will  begin  with  a  need  for  some 
bacsqrounq  information  on  the  oroblem  at  rang.  Has  there 
been  a  oroblem  like  tris  oe  f  ore?  How  have  the  people 
involved  reactoo  in  similar  situations?  <lre  there  any 
Deoole  available  who  have  experience  in  this  area?  These 
are  all  examples  of  toe  tvoes  of  questions  which  will  need 
to  be  answered. 

?\i  e  x  t »  the  decision  mgker  will  want  all  of  the  latest 
facts  surrounqino  tho  oroblem.  oho  is  available  to  move 
into  the  area?  l'ow  1  ora  will  it  take?  .Vno  is  there 

already?  ./hat  forces  does  the  oooosinc  sine  nave  available? 
A hat  is  haooening  now?  These  and  many  more  are  the 
questions  Surrounding  the  current  situation. 

Mow#  the  decision  maker  will  want  to  know  what  the 

ootions  are  and  what  the  adversaries  oot ions  are.  In 

addition,  will  be  the  need  to  know  of  any  activities,  either 
in  process  or  oendino,  which  mav  nave  an  impact  on  the 

decision,  \/ext  is  the  assessment  of  the  probabilities  of 
tne  oossiole  outcomes  of  these  activities.  The  criteria 
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which  will  be  uSPft  in  analyzing 


t  h  e 


a) f ernat i v«s  must  be 


listen  and  relative  i^oortanco  weionts  tor  these  determined. 
c i ra 1 1 v  toe  decision  m  3  <  a  r  evaluates  the  alternatives  and 
selects  a  course  of  action  to  be  followed. 

How  does  the  decision  maker  oo  all  t h f s »  esoecially 
when  the  i reacts  of  a  wrong  decision  can  be  so  costly?  It 
certainly  can  not  be  done  alone.  The  decision  maker  must 
relv  on  *-he  aid  of  ^ev  staff  members  who  have  certain 
expertises  in  soecific  a  r  »  a  s  .  but  even  with  cpmoetent 
staffs/  the  task  is  often  too  oreat.  Cotouterizeo  decision 
aids  may  h  e  the  only  answer. 

There  are  several  reasons  why  comouterized  decision 
aids  would  oe  useful  in  the  C3  decision  ma*ina  process 
fallen  ef  al.»  lb7t>,  oo.  ).  These  reasons  are: 

1  .  It  is  an  effective  tool  for  imeroving  the 
dialonue  a^onci  staff  officers  who  are  workina 
tooefher  on  a  particular  problem.  It  would  be 
helpful  in  eliminating  misunderstandinos. 

2 .  The  process  of  using  a  decision  aid  dives  staff 
officers  the  opportunity  to  include  what  they 
considers  important  in  the  analysis.  Then  they 
can  see  how  these  factors  actually  impact  on  the 
solution. 


3. 

It  is 

very  easy  to  evaluate 

several  courses 

o  f 

act  ion. 

Variables  which  are 

uncertain 

can 

be 

C  handed 

to  a'low  for  the  best 

and  worst 

cases 

i  n 

the  analysis 


'4  .  D  r  i  t  i  c  a  1  area?  can  noa  r,s  *  o  c  u  s  e  O  on.  ull  too 
often*  too  T'jch  tine  is  wasted  in  o  i  s  c  u  s  s  i  n  u  or 
arouino  -rat  may  se*r  to  he  imoortant  point?/ 
hut  turn  out  to  Oe  rather  iosionificant  in  the 
final  solution. 


5  .  !  iTe  new  he  in  your  favor  rather 

ooconpnt.  »' o S t  oeoole  oo  not  react 

uri°r  /-i-re  constraints.  Peon  1  e  tend 
react  to  uniToortant  events  an n  make 
hasty  judOTents.  Decision  airs  enable 
the  decision  raver  to  follow  a 
routine.  Tnon  the  aid  performs  the 
calculations  in  a  real  tine  manner, 
additional  t  i e  to  either  wait  for 
infornation  or  actively  seek  for 
analysis  shows  a  need. 

There  are  mjt?rous  examples  of  C3  rr 
comuteri;ei  decision  aids  would  6 e  useful  ano 
are  sore  examples: 


than  you  r 
well  w  n e  n 
to  o  v  e  h 

au i c  k  and 
ann  force 
systemat i c 
necessary 
This  dives 
addi t i onal 
it  if  the 


obi ers  where 
the  following 


1.  Decision  aids  can  and  are  beino  used  to  monitor 
earlv  warnino  systems  for  any  indications  of 
i-roendino  crisis.  It  is  important  to  net  allow 
events  to  control  usz  but  rather  to  ocssiblv 
have  us  share  the  events  (Andriole/  19  8  0/  o. 
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?.  A  p9co,it  example  cnf'capn?'i  the  analysis  of 
ii f  f oronf  evacuation  postures  that  a  commander 
was  consioerino  in  the  face  of  uncertainty  about 
a  develooinn  crisis  that  could  Have  maae  it 
necessary  to  evacuate  U.S.  nationals  from  a 
t  h  i  pH  country  (  i  1  1  i  a m  S  /  1Q7B,  r>.  <?  7  )  . 

3.  Tr>e  nomal  contingency  olanninq  process  that 
constantly  takes  olace  is  another  e»anole 
f.iiMiam  s#  lq7B,  o  .  2  B )  .  The  decision  aid 

allows  vou  to  easily  o  1  a  v  the  "what  if"  damp. 

Training  is  another  e*amole.  by  allowino  staff 
officers  the  chance  to  use  the  a  ’  d  s  based  on 
oast  crises/  they  become  familiar  with  current 
plans  ana  opocoHures  and/  if  realistically  used/ 
recome  used  to  ooeratim  in  a  crisis  environment 
(Allen  ef  a  I  .  /  1  Q  7o  /  p .  b )  . 

5.  Decision  aids  may  oe  used  for  establishing 
policy.  Options  can  be  evaluated  and  tested 
prior  to  deciding  on  firm  rolicy. 

0.  Toe  design  o*  new  C3  systems  would  be  another 
.*.  h  e  n  e  v  e  r  new  systems  are  oesioned/ 
there  is  always  a  tradeoff  of  capabilities 
bee  ausa  of  insufficient  funoino.  These  aids 
would  heln  determine  the  safest  cuts  io 
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in.  f  x  r  s  t  tng  c  3  dec  i  s  t  on  at  os 


This  chaoter  describes  several  C3  corouter-basea 
information,  decision,  and  forecasting  systems  designed  to 
lin<  the  human  user  with  the  information  so  critical  to 
efficient  informal  ion  manaoement. 

Tnere  presently  exists  much  too  little  emohasis  uoon 
tne  content  and  esoectally  the  form  of  the  information  and 
data  that  flow  thrguah  C  3  svstems  (Andriole,  1^80,  d.  3). 
Toe  questions  of  how  the  information  shoulg  aoDear  to  the 
user,  how  the  information  should  De  stored,  retrieved,  and 
manipulated,  a  n  g  how  f  h?  ever  increasing  amounts  of 
information  can  toe  comorehended  by  the  user  have  not  been 
adeauai'ely  dealt  with  ur>  to  this  point.  The  three 
comouter-oased  information  systems  described  below  were 
developed  with  t^esa  Questions  in  ming.  Each  is  aimed  at 
he'oini  t  h »  C  5  svst-n  users  deal  with  the  enormous  amount  of 
information  for  which  they  are  resoonsifcle.  The  systems 
were  developed  with  the  human  user  in  mind  and  thus  exoloit 
and  Supplement  existing  human  information  processing 
c  aoab i 1 i t i es . 

Snatial  Oata  Hase  '■’anagement  “  The  use  of  normal  data 
base  manaoement  systems,  from  the  Perspective  of  the 
occasional  user,  are  difficult  and  inefficient  (Andriole, 
1°80,  •  ^ 1  •  ‘Recently,  research  has  produced  a  prototype 
sratial  data  management  system  whicH  enables  users  to 


hierarchically  store*  retrieve*  and  manioul ate  data  through 
the  USe  of  so  a  tie!  controls  C-nnriole*  l  9  b  0  ,  c.  4  1  .  These 
soatial  controls  allow  the  user  access  to  data  without  the 
nee  3  or  u  s  °  of  the  conventional  keyboard.  Instead*  the 
soatial  controls  permit  access  to  data  using  numerous 
navaoational  aids*  such  as  color,  location  (in  an  electronic 
workspace  projected  on  a  large  screen  display),  touch,  and 
sound.  Specifically*  toe  user  nay  store  and  retrieve  the 
data  according  to  where  he  or  she  finds  it  rost  easy  to 
access  . 

The  soatial  daf  a  management  system  was  develooed  in 
accordance  with  the  way  humans  normally  Store  and  retrieve 
information.  For  examole*  in  a  normal  office  environment* 
ceoole  store  information  in  familiar  ol aces  according  to 
freouencv  of  use,  imcortance*  shaoe*  size*  ana  so  forth. 
The  orototvne  soatial  data  base  mgnaoement  system  uses  these 
osycholoaical  oredisoositions  by  allowing  the  user  to  create 
nis  or  her  own  osycholoaical  workspace  and  project  it  onto  a 
large  disolav.  This  oisnlav  then  is  interactive  ana  caoable 
of  storinq  all  tvoes  of  data  to  include  numeric* 
ohotooraohic*  and  even  audio.  T  t  is  also  easy  to  update. 
The  user  is  then  able  to  manioulate  and  retrieve  data  by 
moving  thro  u on  the  worksrace  with  joysticks  or  touch 
sensitive  disolav  panels.  Figure  7  shows  the  hierarchical 
data  organization  principle.  Figure  #  shows  a  user 
oositioned  in  front  of  a  soatial  data  base  management 
system.  Figure  0  shows  the  joystick  and  touch  panel  mounted 


FIGURE  7 


DATA  ACCESS  CONTROLS 
(Andriole,  1980,  P.  6 
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in  the  arm  of  a  user's  chair.  Data  is  accesses  by  touching 
a  spec i H  c  point  in  the  wor^soace  or  the  touch-sensitive  pad 
ano/or  ryy  m  o  v  i  n  g  the  joystick  forward  or  cackwarg  (to  ascend 
or  descend  into  rKe  ‘'iecarctical  ly  organized  data  case). 

A  (active  C  3  Tnf ornat i on  Selection  -  In  resoonse  to  the 
ever  increasing  information  reaui  ren>ent  s  olaced  upon 
intelligence  officers  responsible  for  analyzing  ana  routing 
information,  toe  a lart i ve  Information  Selector  (ATS)  was 
developed  (  ’adni  et  a'.,  1 0  7  »  o.  1-5).  Durina  this 
development  process,  it  .a;  determined  that  "new  C  3 
tecnnidues  are  reguireo  to  control  information  flow  so  as  to 
best  Tatch  svstem  capability  with  human  characteristics  in 
the  man-comouter  interaction". 

The  a  I  S  process  is  as  follows.  The  system  user  is 
calibrated  during  a  training  ohase  to  determine  how  and  why 
some  ressaaes  (cieces  of  information)  are  selected  for 
further  analysis  and  some  are  rejected.  T  h is  is  performed 
for  each  individual-  user  such  that  re  or  she  has  a 
personalized  information  management  sorting,  routing,  and 
aueuing  system. 

what  happens,  in  effect,  is  the  computer  monitors  the 
user  and  then  internalizes  how  the  user  selects  and  rejects 
information.  Once  the  training  program  is  accomplished,  the 
computer  emulates  the  user  and  automatical  1 y  selects, 
routes,  and  Queues  the  information.  Thus  the  A  I  s  may  be 
viewed  as  an  information  management  assistant  canafle  of 
relieving  its  superior  from  so-e  of  the  burdens  of 
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infer  ration  overload 


Tne  ♦unctions  a^n  elements  n  *  an  MS  are  presented 
Graphically  in  F  i  a  u  r  *>  ID. 


Training  Channel 


FIGURE  10:  ADAPTIVE  INFORMATION  SELECTION 

SYSTEM  CONCEPT  (Andriole,  1980, 

P  .  8  } 

Tne  MS  has  been  developed  for  use  where  C3  information 
is  already  in  an  electronic  form.  The  A  I  S  is  designed  to 
suoccrt  areas  which  are  already  complicated  wit*  too  "uc" 
incoming  information  for  the  human  user  to  handle.  The  AI3 
is  merely  a  computer  orooram  M K  i  c  n  nrovides  helc  to  tne  C  3 
system  user. 

Ultra-daoid  W e a d i n a  -  Fyon  if  systems  such  as  soatial 

3* 


data  case  n^n^oTi^nt  am  MS  can  a  u  i  c  <  1  v  orovi  1?  access  to 
neeie'J  inf  o  r  n  a  t  i  o  n  ,  n  n  e  user  TU?t  still  read  a  no  corGr?heni 
fne  inforiaM'nn  re  fnrp  ti.rtnpr  action  can  be  contenplateri. 
Sott3eou**ntly»  r^searcn  has  n»en  rerforfai  to  irornve  the 
soeed  with  which  users  can  read  and  comorenenrl  (Andriole* 
1980,  oo.  9_ii). 

A  technique  known  as  r  a  o  i  d  serial  visual  presentation 
( 9  S  V  P )  has  c<“en  ^eveloned  and  conouteri  zeo  (or  use  in  C3 
s  vs  t  eTs .  Tha  te*t  is  rrpsenteo  one  worn  at  a  tine  on  a 
cathode  rav  tune  (CM)  in  rani  i  succession  as  Fioure  11 
suggests. 


FIGURE  11:  ULTRA-RAPID  READING 

(Andriole,  1980.  P.  101 


Research  has  imicaf  eo  fiat  single  y  S  V  p  sentences  can 
he  raa<]  ani  acrunto]  y  recall*")  when  s  r  o  w  n  at  a  rate  as  high 
as  Id  words  per  second,  i  cn  is  twice  as  fast  as  people 
normally  r  *  a  d  f  p  ->  r  t  e  r  e  t  a  1  .  ,  \  ''-‘7°  ,  o  .  1). 

heading  sneej  and  c omp r e r p n s i on  is  a  function  of  the 
time  it  takes  for  the  eves  to  transmit  to  the  brain  and  the 
brain's  abilitv  to  process.  Tne  ultra-ranio  reader  was 
developed  to  shorten  to*  time  n  *  t  w  e  e  n  the  eves  and  the 
brain.  Ft  ones  tiis  ov  reducing  eve  movement  to  a  bare 
minimum.  T w e  eves  no  not  need  to  move  back  ana  forth  or  uo 
3 n o  down,  as  is  normally  done*  since  the  text  a c Dears  on  the 
CFT  in  the  samp  location.  In  addition,  substituting  some 
oictures  or  symbols  for  w o r 0 s  has  increased  comorphensior. 
however,  as  with  M3,  tie  text  must  ne  in  electronic  form  to 
be  read  ultra-raridlv. 

The  preceding  three  research  designs  were  directed 
toward  imorgyind  the  information  flow  to  the  decision  maker. 
The  next  three  ex  a moles  are  comouteri2ed  decision  aids  which 
are  designed  to  be  used  primarily  during  crisis  situations. 
Each  permits  the  user  to  se’ect  and  retrieve  history  on  oast 
crisis  situations.  The  best  wav  to  describe  each  is  by 
providing  a  sample  of  the  tyces  of  Queries  which  are 
allowed.  The  first,  as  descriced  in  Figure  12,  enables  3 
user  to  retrieve  and  prescribe  crisis  actions  (or  responses) 
an-i  objectives.  The  second  aid,  as  shown  in  Fioure  15, 
enables  a  user  to  focus  on  the  management  crohlems  which 
occurred  during  oast  crises. 
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The  third  aid  enarles  a  user 
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THIS  SECTION  IS  DESIGNED  TO  ASSIST  DEPARTMENT  OF  DEFENSE  PERSONNEL  IN 
EVALUATING  PROPOSED  COURSES  OF  ACTION  AND  SETS  OF  U.S.  OBJECTIVES  BASED 
ON  DATA  FROM  101  CRISES  INVOLVING  THE  UNITED  STATES  BETWEEN  1966-1976. 

THE  USER  IS  GIVEN  THREE  LEVELS  OF  ANALYTIC  ASSISTANCE 

1.  THE  CAPACITY  TO  SEARCH  FOR  HISTORICAL  CASES  WITH 
SETS  OF  USER-SPECIFIED  U.S.  ACTIONS  OR  OBJECTIVES. 

2.  THE  CAPACITY  TO  IDENTIFY.  ACROSS  ALL  CASES. 

THOSE  ACTIONS  THAT  HAVE  HISTORICALLY  BEEN 
MOST  STRONGLY  ASSOCIATED  WITH  EACH  OBJECTIVE 
SELECTED  BY  THE  PROGRAM-USER; 


3.  THE  CAPACITY  TO  IDENTIFY,  ACROSS  ALL  CASES. 

THOSE  ACTIONS  THAT  HAVE  HISTORICALLY  BEEN  MOST 
COMMONLY  ASSOCIATED  WITH  SETS  OF  U.S.  OBJECTIVES 
SELECTED  BY  THE  PROGRAM-USER. 

WOULD  YOU  LIKE  TO  SEE  A  SUMMARY  OF  ANOTHER  SYSTEM  SECTION? 

PRESS  *Y*  OR  'N'  ANO  RETURN'. 

Y 

ENTER  T.  'II'.  OR  III'. 

II 


CRISIS  MANAGEMENT  DECISION  AID 
(Andriole,  1979,  P.  16) 


FIGURE  12: 


THIS  SECTION  PERMITS  A  DETAILED  EXAMINATION  OF  MANAGEMENT  PROBLEMS 
ENCOUNTERED  IN  41  SELECTED  CASES  (1956-19761. 


MAJOR  PROBLEM  CATEGORIES  ARE: 

1.  SYSTEM-RELATED  DELAYS  IN  DECISION-MAKING 

2.  SYSTEM/PROCEDURAL  CONSTRAINTS  ON  ACTIONS 

3.  LEGAL  ISSUES  INVOLVED 

4.  RESOURCES  INADEQUATE  FOR  DECISION-MAKING/ACTION 

5.  INTELLIGENCE  FAILURES  AT  DECISION-MAKING  LEVEL 

6.  EMOTIONAL/IDEOLOGICAL  ISSUES  INVOLVED  IN  DECISIONS 

7.  INTERPERSONAL  FACTORS  IN  DECISION-MAKING 

8.  PROLONGED  CRISIS  PROBLEMS 

9.  PROBLEMS  IN  SELECTING  ACTION  PERSONNEL 

10.  CONSTRAINTS  ON  OPERATIONS 

11.  PHYSIOLOGICAL  PROBLEMS  FOR  OPERATING  FORCES 

12.  INFORMATION  FAILURES  BY  OPERATING  FORCES 

13.  FAILURES  IN  TAKING  APPROPRIATE/TIMELY  ACTION 

14.  FORST AT  PROBLEMS 

15.  PR08LEMS  IN  THE  OPERATING  ENVIRONMENT 

16.  GENERAL  PROBLEMS  IN  CRISIS  PLANNING 

17.  GENERAL  PROBLEMS  IN  CRISIS  HANDLING 

18.  GENERAL  PROBLEMS  IN  CRISIS  TIMING 

WOULD  YOU  LIKE  TO  SEE  A  SUMMARY  OF  ANOTHER  SYSTEM  SECTION? 

PRESS  Y'  OR  N'  AND  RETURN'. 


Y 

ENTER  T.  *H\  OR  'III*. 
Ill 


FIGURE  13: 


CRISIS  MANAGEMENT  PROBLEM  ANALYZER 
(Andriole,  1979,  P.  17) 
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THIS  SECTION  PROVIDES  ACCESS  TO  INFORMATION  ON  307  CRISES  (1346- 1S76). 

THE  DATA  MAY  BE  SEARCHED  FOR  CASES  MATCHING  THE  USER  S  SPECIFICATIONS. 
OR  A  FULL  DESCRIPTION  MAY  BE  PRINTED  FOR  ANY  SELECTED  CASE. 

CATEGORIES  OF  INFORMATION  COOED  FOR  EACH  CRISIS  ARE: 

•  YEAR  ANO  BRIEF  DESCRIPTION 

•  LOCATION  OF  CRISIS 

•  'NATURE  OR  PRE-CRISIS  ACTIVITY 

•  DURATION  OF  PRE-CRISIS  PERIOD 

•  SCOPE  OF  CRISIS  (DOMESTIC  OR  INTERNATIONAL) 

•  NATURE  OF  CRISIS  (MILITARY, POLITICAL. BOTH) 

•  CRISIS  DURATION 

•  TIMING  OF  CRISIS  RESOLUTION 

•  CRISIS  OUTCOME 

•  ANTICIPATION  OF  CRISIS 

•  DEGREE  OF  THREAT  TO  U  S.  INTERESTS 

•  TIMING  OF  THREAT  DEVELOPMENT 

•  TIME  AVAILABLE  FOR  OECISION 

•  SIZES  OF  PARTICIPANTS/DEGREE  OF  U.S.  INTERESTS 

•  U.S.  RESPONSE /PARTICIPATION 

•  U.S.  OBJECTIVES 

•  NUCLEAR/NON-NUCLEAR  IMPLICATIONS 

WOULD  YOU  UKE  TO  SEE  A  SUMMARY  OF  ANOTHER  SYSTEM  SECTION? 

PRESS  Y'  OR  *N‘  ANO  'RETURN'. 

N 


FIGURE  14: 


CRISIS  DESCRIPTOR 
(Andriole,  1979,  P.  17) 
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0  u  r  i  n  g  crisis  situations*  decision  makers  strive  to 
react  swiftly,  ^eci  "’p  wisely,  ano  cnmmuni cafe  accurately. 
Inis  p»ijui  res  close  coordination  * i  t  h  thei  t  staffs  and  toe 
arility  to  overcome  certain  obstacles  such  as  pressures  from 
tire  constraints*  artiaui ty  of  goals*  and  tne  ronorol ization 
of  tine  with  information  collection.  Some  of  the  problems 
of  information  collection  nave  been  solved  or  at  least  aided 
bv  the  use  of  one  or  -ore  o *  tne  previously  discussed  aias. 
:i h a t  the  decision  -a<er  now  needs  are  effective  decision 
strategies  that  i-oose  rigor  an g  orovioe  a  logical* 
structural  framework  to  assist  them  in  tne  process  of 
choosing  an  optimal  decision  alternative  in  the  face  of 
voluminous  and  often  inconclusive  evidence. 

Orerat.  ions  and  Intel liaence  f  0  P I  T  )  is  a  decision  tool 
that  orcvides  just  such  a  framework  for  deliberation* 
reasonina*  and  analysis  (Amey  et  a  1  .  *  19  7  9*  p.  5 )  .  0  P I N  T 
is  an  on-line*  interactive*  real-time  model  which  aids 
decision  makers  by  prescribing  a  straig^rforwara  normative 
procedure  f3f  organising  an  i  analyzing  difficult  decision 
oroble-s.  These  Problems  may  involve  both  uncertainty  about 
the  outcome  of  future  °vents  and  perplexity  about  the 
complex  value  trade-offs  involved  in  the  choice  of  a  course 
of  action. 

GDIdT  is  a  dec i S i on- ana  1 v t i c  based*  computer  assisted 
decision  aid.  Its  orimarv  objective  is  to  provide  decision 
maxers  a  procedural  frameworx*  or  decision  template*  t  n  a  t 
insures  topir  ultimate  decision  choice  is  a  coherent  c  n  .  A 


coherent  c n o i c o  is  one  which  is  consistent  with  their  own 
value  structures  and  beliefs  about  the  relative  livelihoods 
o  *  ♦  u  t  u  r  e  events  that  will  impact  the  decision  outcome  (  A  m  e  y 
e  t  a  1  .  *  1  7  ,  n .  5 )  . 

The  funsa^ehtal  oroduct  of  QPINT  is  a  co^nuter-stored 
conceptual  reoresentst ion,  or  decision  model ,  of  the 
decision  problem  at  hand.  /<  h  e  r  e  a  s  decision  analysis 
orovioes  the  theoretical  background  and  procedural  guidance* 
the  OP  I  **•(  T  decision  model  provides  the  snecific 
methodological  tool  for  orocessino  information  and 
evaluating  the  various  decision  alternatives  ooen  to  the 
decision  Tatar.  in  in- depth  explanation  of  how  OP  I  NT  aids 
in  the  dec i si  on  process  is  discussed  in  Charter  Four. 

Evaluation  (  E  V  A  L  )  *  another  on-line*  interactive* 
real-tiTe  model*  incorporates  another  decision  strategy 
which  provides  the  decision  maker  with  a  normative  procedure 
for  analyzing  difficult  decision  pronlems.  It  is  especially 
designed  to  evaluate  and  comnare  possible  options  using  a 
mu  1  t  i -a  1 1  r  i  nu  t  e  utility  analysis  techninue.  E.VAL.  is  best 
applied  to  procurement  decisions  wherein  systems  are 
compared*  but  is  appropriate  for  a  diverse  set  of  decision 
analytic  problems  (Decisions  and  Desions*  Inc.*  1Q77,  po. 
1-5)  . 

The  use  of  E  V  A  L  is  broxen  into  t»o  phases.  First*  the 
decision  maker  creates  the  structures  of  a  hierarchical 
multi-attribute  evaluation  model  using  the  tVAL  Structure 
program.  Then,  this  model  is  evaluated  and  processed  using 


as 


the  E  V  A|_  p  r  o  a  r  a 


In  order  to  utilize  tne  EVAL  Structure  p  r  o  o  r  a  m  ,  the 
nro!-.)?*  "ujst  be  decomposed  i  <■>  t  o  a  hierarchical  structure 
which  reflects  t  ha  loiical  interrelationship  of  all  the 
factors  involve--!.  This  structure  is  then  enteteo  and  the 
erode  1  is  cheated.  Now  under  the  direction  of  the  EVAL 
proaram,  the  criterion  used  for  evaluation  the  alternatives 
are  entered.  f'nce  entered*  their  respective  weiahts  of 
importance  are  entered.  u’xt,  the  relative  score  which  each 
alternative  achieved  for  each  criteria  is  entered.  For 
example*  consider  three  radio  sets  which  are  being  evaluated 
for  selection  before  procurement  (Al,  A2,  and  A3).  The 
criterion  of  evaluation  are  cost*  range*  portability*  and 
reliability.  Each  of  these  criteria  are  weighted  30%,  20% * 
10%,  and  U0%  rasrect i vel v,  Next,  tne  three  radio  sets  are 
evaluated  accnroino  to  each  criteria  using  an  arbitrary 
utility  scale.  Cost  would  show  scores  of  5  0  for  set  Al,  <10 
for  s»t  A2*  and  b 0  for  set  A3.  This  means  A3  is  considered 
to  have  the  best  cost.  This  process  is  repeated  for  each 
criteria.  It  is  also  normal  for  the  criteria  to  have 
sub-criteria  which  would  also  reauire  the  same  procedure. 

Next,  EVAL  gives  the  evaluation  scores  *  or  each  of  the 
alternative  systems  resulting  from  the  aooreqations 
prescribed  in  the  model  structure.  Now  the  decision  maker 
can  examine  the  effect  of  the  criterion  weiahts  throuah  a 
sensitivity  analysis.  This  will  auickly  show  him  which 
criteria  are  the  critical  ones  and  take  action  to  insure  the 
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nafa  surroundinp  that  criteria  is  as  accurate  as  possible. 
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arid  feel  confident  about  the  decision  re  must  make. 

Another  decision  aid  has  been  developed  to  assist 
orouos  in  decidina  how  to  leal  with  imoortant  issues  or 
urntile’S.  Its  ruroos?  is  tre  improvement  of  decision  makinq 
by  coot  i  ''ubus  participant  interaction  with  the  comouter 
durinq  tre  decision  "aki'ia  process  (Leal  et  al.*  1478*  d. 
l-l).  -'embers  of  fn»  orouo  are  allowed  to  incut  their 
resoecrive  estimates  of  the  occurrence  of  soec  i  f  i  c  events 
anq  thoir  values  (weidhts)  of  the  importance  of  specific 
decision  outcomes.  The  decision  aid  then  ccmoares  the  inout 
and  informs  the  orduo  members  of  any  d i s an r eemen t s  between 
them.  d u t  mn3t  importantly,  it  tocuses  discussion  only 
on  t  h  ■=»  disaore^mants  which  are  important  to  the  stated  aoal . 
ire  differences  which  are  not  imoortant  are  shown  to  be  just 
that.  Finally*  it  suaaests  to  the  qrouo  Precisely  how  their 
disaoreements  m i  •  i  h  t  he  resolved.  This  enables  qrouos  to 
move  much  more  ouic^lv  toward  decisions.  As  a  by-product* 
profiles  ar*>  developed  o*  the  participants  which  snow  who 
has  exoert-ise  in  special  areas  as  well  as  who  is  a 
risk-avniier  or  ris^-taker  (Leal  et  al.*  1978,  po.  1-?  to 
1-10).  T  K  e  function  of  *•  n  e  3id  is  shown  in  F  inure  15. 
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FIGURE  15:  THE  GROUP  DECISION  AID  PROCESS 
(Andriole,  1979,  P.  17) 


Tie  decision  o  r  o  u  o  is  composed  of  t  h  p  participants  ( de~ 
;isi  rn  -a  *  e  r  3  )  ?  an  i  nterieli  at  or,  anrl  a  director, 
rhe  interred iatcr's  n  r  i  m  a  r  y  function  is  to  facil- 

j  t  a  t  °  co^'iini  caH  on  between  the  q  r  o  u  o  and  rhe  computer. 
Tris  is  done  tv  translatini  stolen  requests  into  comDut- 
»r  inputs.  These  inputs  c  o  u  1  J  be  lists  of  events?  alterna¬ 
tive  actions?  reauests  for  displays?  or  modifications 
to  any  orevinus  inputs.  The  director  hplns  with  the  outout 
hy  oresentino  and  eol  ainino  it  to  the  cart  i  c  i  cant  s. 
fhe  director  also  focuses  the  q  r  o  u  p  '  s  activities  and  insures 
the  incuts  are  appropriate.  a  s  a  result?  virtually  no 


participant  training  is 


ne(*,1er)  to  ooerate  the  system?  and  a 


arour  can  cecin  *  o  r  k  cn  its  decision  dtodI em  almost 

imneoiately  after*  it  is  convened. 

inntHor  comnut er i zpq  fli.ii  tKe  tarty  ,'iarnjrq  am 
,/|oni?orina  Systs"'  (FdA'S),  has  h»?n  otvtloDen  to  a  i  d  t^e 
Indications  ana  tvarnina  Cli'i)  connuni  ty.  ELAN'S  orovides 
them  with  an  interactive  computer  based  system  of  political/ 
military/  ana  domestic  indicators  for  daily  monitoring  of 
international  and  iotra  national  affairs  (Paly/  19  7  8,  d. 
U  )  .  The  system  n.as  the  capability  to  store  Quantitative 
international  political  data  from  19bfc  for  most  countries  of 
the  world  and  can  therefore  ce  use a  to  ao  retrospective 
analysis  as  -ell  as  current/  d  a  i  1  v  Is.*".  The  data  is 
aooredated  by  month,  Quarter/  and  year.  Ev.AMS  has  the 
canahility  to  track  sinale  countries/  combinations  of 
count ry-pa i rs /  reaions  or  the  entire  world  with  these 
quantitative  indicators/  or  specific  reaions  m  a  y  be 
specified  (Hjl  y,  1  0  7  8 ,  p.  t>).  At  present/  there  is  some 
doubt  as  to  whether  E •’) A S  will  ever  be  fieldea  because  its 
operation  is  extremely  slow.  But  the  concent  is  a  aood  one 
and  further  research  may  prove  beneficial. 

The  EdAMS  is  com.  orised  of  the  following  comoonents 
(Apdriole/  1980,  dp.  ?B-34)t 

1.  General  Scans  -  A  oeneral  scan  is  an  aaqreaat ion 
of  countries  ov  soma  criteria/  e.a./  a  scan 
defined  hv  jeooraohic  region  would  aaarpoate 
countries  hv  arouos  such  as  Latin  America  or 
Middla  Fast arn.  A  general  scan  allows  an 


analyst  to  look  at  several 


countries 


t  axen 


t-nrjetner  r  a  t  m  a  r  tnan  1  o  o  k  i  n  ci  sen  a  rate!  y  at  fany 
ccuntrv-nairs.  It  t*e  indications  tor  tne  grouo 
as  a  .^cl  a  simest  unusually  hint  activity/ 
tension/  or  uncertainty#  t  re  analyst  can  oroD  to 
the  count  rv-by-ccunt  rv  level  or  even  track  tne 
recent  activity  of  a  sinale  country  to  determine 
tne  source  of  the  disturbance  or  the  renional 
level.  r,  e  n  e  r  a  1  scans  thus  ra<e  tne  system  m  u  c  n 
to  re  e  t  f i c i 0  n  t  and  less  tine  consuming  for  the 
ana  1 yst . 

2  .  Quantitative#  Political#  Military,  fcconomic#  am 
Domestic  Indicators  -  Quantitative  indicators 
for  crisis  earning  include  fore i an#  domestic# 
and  international  colitical#  military#  and 
economic  factors.  The  indicators  are  noth 
dynamic  (events- based )  and  static  (attribute 
based)  . 

3.  N'ulti  vethod  forecast  ina  Caoability  -  The 
unified  multi -method  forecastinq  capability 
requires  the  system  to  Generate  different  kinds 
of  forecasts  or  warnings  via  different  methods 
for  different  events  and  conditions..  Sucn 
notions  have  been  desianed  with  reference  to 
Objects#  aoals#  and  methods  of  forecasting. 


fcxreri mentation  is  still  beinc  oerformen  in  this 


arei 


Additional  researc"  is  currently  noinq  rerior-"?')  to 
lesion  an  ail  *  n  i  c  h  will  focus  on  Soviet  crisis  behavior 
since  Vo  rid  qr  II.  This  aid  will  include  the  following 
tasks: 

1.  0?vel  ooi'pnt  of  sn  inventory  of  Soviet  crisis 
'nanaoeneot  pphavior,  l°'-ifc“lQ7Q. 


2.  Collection  of  data  on  the  ciianacteri  sties  of 
these  crises  to  show  t^e  nature  of  Soviet 
military  crises. 

.  A  more  intensive  analysis  and  codino  of  key 
Soviet  crises  to  identify  the  crisis 
pn v i r rnT» n t s  that  mav  affect  the  occurrence  of 
crisis  r-anaaenent  crorlems  encountered  by  the 
Soviet  union,  and  variables  describinn  crisis 
objectives,  actions,  and  results  for  the  Soviet 
Uni  on  . 

<J .  Statistical  analys®s  of  the  characteristics  of 
Soviet  crisis  t a n aoemen t  ODerations,  the 
“nvi ronmwnt  in  which  they  have  taken  olace,  and 
the  crisis  'T-anadP'iipnt  onrlsTS  encountered  by 

the  SovlPt  1  n  i  On, 


u.s 


■ill  ot  t  ‘'ese  lec  i  s  i  on  ?iiis  Kave  t'een  aavert  i  s?d  as 
oeina  va  i  mn  I  e  to  i“r  i  si  on  ’idkim,  M  o  n  e  Mve  been 
exceri^antallv  evaluated  *i  tt  knc^leoaeable  subjects  in  a 
decision  m a «. i n q  environment.  4  baseline  of  such  evaluations 
is  imoortant  to  estaolish  in  order  to  Drovide  a  useful  tool 
and  better  yeti  to  provide  for  much  needed  technology 
transfer  a n d  real  v  o  r  1  d /  real  time  feedback  on  the  value 
(utility)  of  toe  tecnnolonv. 

0  P  I  i  T  «  a  s  selected  as  an  appropriate  candidate  for 
experimentation  tor  a  n u m b e r  of  reasons.  Firsts  it  is 
readily  availaole  as  a  package  at  taval  Postoraduate 

School .  Second,  all  of  the  eauip*eot  required  for  its 
ocerat ion  already  exists  at  the  School.  Finally/  due  to 
boto  time  and  monetary  constraints/  the  procurement  of  other 
aids  •*  a  s  unfeasible. 
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I  7.  p  1 :  ‘  •  C  T  T  J '  ■  1 L  OFSCh-IPT  IP»i  UF  PPI'IT 


PRIdT  is  a  'ipcisio'i* analytic  based,  computer-assisted 
decision  strateov  (A-nev  ef  al.»1979,  r> .  <4 )  .  It  was 
desianed  and  built  by  decisions  and  Desians  Inc.,  of  '•icLeani 
V i r  d i n i a  *  under  contract  to  the  defense  Anvanced  Research 


Projects  A  o  e  n  c  v 

(DiPPA)  . 

It  was  written 

in  the 

APL 

oroorammina  ] 

a  no  u  a  a e 

for  the  I  P  v  5100 

computer. 

I  n 

addition,  0  p  1 '  i  f 

has  recent 

ly  been  rewritten 

in  FORTRAN 

UP 

for  use  on  POP  11  series  computers. 

The  general  nurnose  of  OpTuT  is  to  aid  decision  makers 
Ov  providing  then  a  capability  to  constructs  Stores 
retrieves  exercises  and  refine  nec  i  s  i  on-ana  1  y  t  i  c  models  of 
Comdex  decision  problems  dev  face.  The  OR  I  NT  decision 
models  which  is  interactives  real -tines  and  on-line,  is  an 
orqanizinq  framework  for  information  orocessina.  Decision 
analysis  is  a  methodological  tool  with  which  the  decision 
maker  defines  and  exercises  the  0  R I N  T  model  to  evaluate 
decision  alternatives  oertaininn  to  the  problem  CAmey  et 
a  1 . s  19  79,  oo.  3-?l). 

The  Up  I  N  T  system  is  desianed  to  be  used  interactively  bv 
decision  makers  who  are  relatively  unsophisticated  with 
respect  to  computer  technoloov.  Accordingly,  the  design 
satisfies  two  human-factors  objectives:  PRINT  is  a  menu- 
driven  system  and  is  o  e  s  i  o  n  o  i  to  be  foraivina  of  o  r  o  ~ 
ce dural  errors  rv  the  user. 
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E^ch  iec  i  s  i  rn  model  created  rv  the  user  nav  be  oiven  a 

uniaue  labels  a  I  t-  ‘lnuqn  each  is  constructed  us  i  no  the  same 

aonen'!  format.  f  no  model  format  is  ^o»n  uracn  i  ca  I  1  v  in 
fnupn  In. 

In?  lorfat  consists  c*  the  fol lowing  elements 

chichi  «hen  co",c)?fplv  sdpc  i  ti  en,  uniauelv  c»  M  ne  an  0  P  I N  T 

decision  undo  1  ; 


i  . 

[no  Ipc i 

si 

-  A  short 

label  oefinino  the 

decision 

Or)K 

Ion.  This  label  is  also  aooliel  to 

r  n  e  o  e  c  i 

c 

0 

in 

-coel  ano 

is  used  to  store  and 

retrieve 

t  re 

m  n  de  1  • 

<2. 

0 e c i si  on 

A  1 

•prn^rives  ~ 

A  list  of  decision 

a  l  t  e  r  n  a  t  i 

v  e  s 

available  to 

the  decision  maker. 

Earn  a  1  *■  e 

i  ve  is  appropriately  labeled. 

3. 

An  1  j  n  c  e  r  t 

a  i  0 

future  Event 

-  A  kev  uncertain 

event  # 

E  , 

that  will 

influence  the  eventual 

outcome  of  t  h 

e  decision. 

The  uncertain  event 

is  attached  to  each  of  the  decision  alternatives. 

a .  Event  Outcomes  -  A  list  of  the  discrete  event 
outcof»S/  each  a nr r o o r i 3 t e 1 y  labeled^  that 
too ether  define  the  universe  of  oossibilities 
reoardinc  the  occurrence  of  the  future  event. 

5.  Event  Probabilities  -  A  vector  of  orobahilities 
that  a^e  associated  with  the  event  outcomes* 
which  reoresent  the  probability  that  an  event 
■will  occur.  A  crobabilitv  is  a  measure  of 
uncertaintv.  It  is  a  number  between  0  and  !» 
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inclusive/  t  h  a  t  represents  fho  extent  to  w  H  i  c  in 
an  in  !i v i  Tja I  h» 1 i eves  a  future  event  will  occur. 
Mowever,  in  tnj«  s  o  e  c  i  f  i cation/  oroo abilities 
are  expressed  as  a  oercentaoe  of  certaintv; 
e  .  a  .  /  as  4  0  vice  0.4. 

b.  Decision  Outcomes  -  The  elements  discussed  thus 
far  ci»f  in?  Possible  decision  outcomes.  Each 
decision  outcome  is  *  o a i r e d  combination  of  one 
decision  alternative  'with  one  event  outcome. 

The  remaining  three  elsnents  of  the  model  format  are 
used  to  soecifv  the  relative  consequences  associated  with 
the  decision  outcomes.  The  consequence  of  an  outcome  is 
exoressed  in  t  °  r  ~  s  of  the  relative  rearet  that  would  be 
exoerienced  bv  t h  e  decision  maker  should  toe  outcome 
actually  occur.  These  three  elements  are: 

1.  Decision  Outcome  Criteria  -  4  list  of  criteria/ 

each  Aoorooriatelv  1 aoel ea>  bv  which  the 

decision  a  k  e  r  would  jucoe  the  relative  rearet 

associated  with  the  decision  outcome. 

?.  Criteria  Neiqhts  -  -V  vector  of  weiqhts 

associated  with  the  criteria/  which  represent 
the  relative  contribution  of  the  criterion. 
Criteria  w  e  i  a  r>  r  s  are  «  x  p  r  e  s  s  e  d  numerically  as  a 
oercentaoe  of  the  whole/  e  .  a  .  f  as  6  0 X . 

3.  Reoret  *  Regret  is  a  measure  0f  the  conseauence 

of  a  decision  outcome.  The  total  rearet 


assigned  to  a  decision  outcome  is  a  weighted 
linear  combination  of  toe  individual  criteria 
regrets.  For  each  criterion  ana  tor  each 
decision  outcome/  the  user  must  specify  a  value 
of  rearet.  1  r  e  a  r  e  t  is  a  number  between  0  and 
-1/  inclusive/  that  reoresents  the  relative 
degree  of  dissatisfaction  that  the  decision 
m  a  *  e  r  associates  with  a  particular  decision 
outcome.  Z«ro  represents  no  rearet 1  -1 
represents  maximum  regret.  However/  in  this 
specification  reorets  are  always  expressed  as  a 
oercentaoe  of  the  maximum;  e  .  a  .  /  as  -30  vice 
-0.3.  Refer  to  Fiaure  17. 

This  completes  the  model  format.  Tne  decision  m0a,»i  is 
completely  and  uoiauely  specified  when  the  elements 
described  above  are  defined  bv  the  user.  The  input 
soeci fications  oescribinq  the  model  can  be  process® a  to 
produce  th®  followino  results! 

1.  Comoineo  Value  Re  a  ret  Matrix  -  A  single  matrix 
that  displays  the  total  or  combine  n  rearet 
associated  with  each  of  the  decision  outco-es. 

For  each  outcome/  the  combined  regret  is 
obtained  bv  weight  i  no  and  aadina  the  comrcr'e,'t 
rearets  contributed  by  each  criteria.  A  combined 
value  rearet  matrix  is  shown  in  Fioure  lb. 

2 .  Eire c  t  ®  d  Value  v  a  t  r  i  x  -  A  Single  matrix  that 
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FIGURE  17: 


REGRET  MATRICES 

(Amey  et  al.r  1979, P.9) 


•  f 


COMBINED  VALUE 


Ex  (LABEL) 

E2  (LABEL) 

E5  (LABEL) 

S1 

(LABEL) 

0 

-17 

-31 

d2 

(LABEL) 

-9 

-9 

-17 

d3 

(LABEL) 

-24 

-22 

-22 

d4 

(LABEL) 

-30 

-26 

-26 

FIGURE  18:  A  COMBINED  VALUE  REGRET  MATRIX 


d  i  s  c  1  a  v  s 


the 


w  e  i  a  h  t  e  a 


e  «oec  t  ed 


rea  r  e  t  , 


associated  with  each  decision  outcome.  The 
expected  value  matrix  takes  into  account  the 
relative  livelihoods  of  the  event  outcomes. 

5.  Expected  value  Vector  -  A  vector  that  aisolays 
the  weighted  expected  rearet ,  associated  with 
each  of  the  decision  alternatives. 

Tho  exoecteo  value  matrix  a  n  o  t.  he  expected  value 
vector  (total)  are  d  i  s  o  1  a  y  e  d  together,  as  shown  in  Figure 
1°. 

There  are  two  sensitivity  analyses  that  are  useful  to 
the  user.  Both  are  based  on  the  exoecteo  value  vector  and 
are  described  as  fo'lows: 

1.  Threshold  a  t  r  i  x  -  A  matrix  that  disolavs  the 
eleven  exoecteo  value  vectors  aenerated  bv 
either  one  of  the  following  operations,  at 

the  user’s  ort  ion 

a.  Varying  the  orcbability  of  a  designated 
event  outcome,  from  0  to  100%,  in  steps  of 
10.  The  other  event  outcomes*  maintain 
their  pronortional  relationships  with  each 
other. 

b.  Varying  thp  weioht  of  a  designated  criterion 
from  0  to  1  0  d  *<  ,  in  steos  of  10.  The  weights 
of  the  other  criteria  maintain  their 
orooortional  relationship  with  each  other  in 


b  0 


EXPECTED  VALUE 


Ex  (LABEL) 

E2  (UBEL) 

E3  (LABEL) 

TOTAL 

D1 

(LABEL) 

0 

0 

-4 

-4 

D2 

(LABEL) 

-5 

0 

-2 

-7 

d3 

(LABEL) 

-12 

-1 

-3 

-16 

d4 

(LABEL) 

-15 

-1 

-4 

-20 

FIGURE  19:  AN  EXPECTED  VALUE  MATRIX  AND 

AN  EXPECTED  VALUE  VECTOR  (Amey 
et  al.  ,  1979  ,  P.  12) 


6  1 


the  same 


manner 


as  do  the  orooab i 1 i t i es 


lascrinpti  in  the  nreceding  citaorao"  • 

[n  rotn  of  tho  afo«e  cas»5i  the  least  rennet  li sol avac 
in  the  e  x  oec  t  eg  value  oavoff  vector  is  identified  dv 
an  asterisk.  Normally*  the  decision  alternative  that  leads 
to  the  decision  outcome  having  the  least  rennet  will  change 
as  the  lesianateo  event  orobability  or  criterion  weight 
is  incremented  f  row  0  to  1 0  0  X  •  The  coints  of  change 
are  referred  to  as  threshold  mints  and  are  noted  on  the  ma¬ 
trix.  Figure  ?<)  shows  an  examole  threshold  matrix. 

EXPECTED  VALUE  WHEN 
PROBABILITY  OF  Ex  (LABEL)  IS: 

0  10  20  30  AO  50  60  70  80  90  100 
Dx  CUBED  -57  -52  -46  -40  -34  -29  -23*-17*-ll*  -6*  0 

D2  (UBEL)  -51  -47  -42  -38  -34  -30  -23*-21  -17  -13  -9 

D3  (UBEL)  -27  -27  -27*-26*-26*-26*-25  -25  -24  -24  -24 

D4  CUBED  -26*-26*-27*-27  -27  -28  -28  -28  -29  -29  -30 


FIGURE  20;  A  THRESHOLD  MATRIX  (Amey  et  al . , 
1979,  P.  15) 
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?.  Manual  1  v  ci-nae  event  DPohaoi  1  i  ties  -  The  user 
ngv  oenerate  -a  fesf  exoected  value  vector  that 
is  nas°o  on  an  ant i >rari lv  assigned  vector  of 
event  rrohahi I i ties.  The  user  nay  scecifv 
several  different  probability  vectors  and  note 
the  resultant  exoecteo  value  vectors.  An 

exarole  of  the  disolav  is  shown  in  Figure  21. 


KOREAN  INTENT 

NO  ACTION  HARASS  SHOOT  DOWN 

CURRENT  LIKELIHOOD:  60  30  10 

ENTER  REVISED  VALUES:  30  60  10 

NEW  VALUES  :  30  60  10 

IF  THESE  VALUES  ARE  CORRECT  TYPE  GO:  (GO) 

EXPECTED  VALUE/REGRET 


DO  NOT  FLY  -38 
MODIFIED  ROUTE  -26 
ARMED  ESCORT  -10 
HIGH  PERFORMANCE  A/C  -10 
NORMAL  MISSION  -29 


FIGURE  21:  MANUALLY  CHANGED  EVENT  PROBABILITIES 
(Amey  et  al.,  1979,  P.  16) 


QP  I’;T  is  desiuneg  to  oerfom  the  basic 


described  below: 

1  .  v  a  i  n  t  a  i  n 
v  a  r i ou S 


a  library  of  0PINT  models 
.decision  no  dels,  filed  by 


functions 


Store 

their 


associated  labels 


Load  an  ex i s  t i no 


DPf'Mr  lorte' 


D  i sol  ay  the 


labels  cl  f  o  c  s  e  models  scored  in  the  no  del 
I  i  r  r  a  r  v  ,  a  n  i  n  e  r  m  i  t  Me  user  to  retrieve  any 
desired  "ioi®l  .  The  loaded  model  is  referred  to 
3S  the  current  -none!. 

Disolav  f  he  results  of  the  current  rode  1  - 

Permit  the  user  to  examine  the  structure  and 
content  nf  tHe  current  model  cy  di sol  avion; 
a.  event  orob. abilities/ 
r  .  criteria  •-  e  i  o  n  t  s  / 

d.  c o n n i n e d  v  a  1  u  “  »atri»( 

e.  exoected  value  matrix,  and 

f .  expected  value  vector. 

Revise  ftp  current  node!  *  Pern  it  tse  user  to 
ma«e  chanoes  to  the  structure  and  content  ot  the 
current  model.  The  user  mgv  revise: 
a.  event  orohan  i  1  i t i es » 
o  .  criteria  and  criteria  w  e  i  d  h  t  S  , 

c.  decision  al t ernat i ves , 

d .  red  re  t  s  «  and 

e.  combined  value  regret  matrix. 

Save  the  current  model  ~  Permit  the  user  to  add 
the  current  mode!  to  the  model  library. 

Perform  sensitivity  analyses  -  permit  the  user 
to  test  the  sensitivity  of  the  current  model  bv 
determining)  thresholds  or  manually  changing 


b 


r  r ori”  i  1  i  t  i ?s  . 

f  -  o  3  ♦  e  4  '  a»  ~  ’  '  T  ''--ol  -  »’  a  r  —  l  f  t  n  p  uS?r  tO 
-  r  j  4  t  f.  .»  ~  i  ,  .“ir1'  r|-?o  t'“cc'r®s  the 

*  ,rror.f  -r  ^  ;  a>  '  .  T~a  ,s&''  creates  a  imp  1  by 
ccec  i  *  v  i  "a  a>la>-iar>rs  reau  i  rpo  ter  r  b  e  ncoel 

as  e»rl  ai  ,'“'i  ea^l  iff.  Tp®  user  pay  also  use  a 
reference  aa^ble  to  v  a  1  i  n  a  t  e  the  assianed  values 
o t  p®jp»t . 
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V 


THE  neSK-'!  OF  THE  E  XPE  9 !  '-’F  jT 


Tre  conduct  of  e<D?ri"ents  u sin--  cumnanr)  and  control 
decision  aids,  such  as  0 P I N T ,  nresent  special  nroolems  (Cain 
am  Pod,  1978,  o.  b).  H  i  s  t  o  r  i  c  a  11  v  ,  the  evaluation  of 
conoani  and  control  systems  ras  been  focused  on  hardware 
'aasuraTopf  a,  Little  a»  font  j  on  was  raid  to  software, 

esoec  i  a  1  1  y  decision  at  is  .  8  s  a  result,  few  well  defined 

ouantitative  Teasure"orts  exist  for  measurino  the  duality  of 
decision  aids  (binai^o,  1°77,  do.  5-b).  Seconn,  there 
are  no  "connect"  answers  to  a  decision  problem  (Daniels, 
1 9 7 7 ,  o.  In).  Rather,  an  aid  such  as  0 P I N T  is  a  cognitive 
tool  to  assist  the  decision  ma<pr  in  reveal  inq  and 
formuli^ino  the  elements  of  the  decision  nroolem  as  the 
decision  maser  oerceiyes  them.  Lastly#  durina  the 
develocment  of  thse  decision  aids,  there  has  been 
relatively  little  svneroism  betwe»n  developers  and  potential 
users  (Cain  and  Pot ,  1978,  o.  o).  Users  are  often  not 

familiar  with  the  new  technoloav  and  are  unsure  exactly  what 
to  exoect  in  the  wav  of  performance.  Developers  are  unsure 
exactly  what  the  potential  users  want.  This  occurs  because 
thev  aren't  sure  who  the  users  will  be  and  causes  a  very 
General  aid  to  be  developed  -  ope  which  will  try  to  suit 
everyone's  needs. 

Any  experiment  w h i  :  h  involves  t  m  e  use  of  decision  aids 
must  be  desioned  carefully.  Tre  exoerimpnt  Should  be 
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desianed  in  ir^er  t  o  assess  doth  the  caoani  )  i  tv  of  the 
techno! o a v  ann  its  overall  utility  as  it  r  e  1  a  t  e  s  to  total 
system  oerfomince  ( h  a  r>  i  e  1  s  ,  19  7  7,  p. .  1  ‘4 )  .  There  are  four 

levels  of  f><roi‘if»nts  v  a  r  v  i  n  q  in  formalist  from  very  loose 
"free  clay"  with  only  Subjective  j  u  d  q  m  e  n  t  outout  to  "hiahly 
structured"  witn  co"f)tete  soec  i  f  i  c  a  t  i  on  of  conduct  of  trials 
and  output  consistinc  of  carefully  measured  system 
attributes  (^rr  et  a  1  . ,  1  Q  7  ^  ,  n.  I)  ,  These  levels  from 
least  structured  to  most  structured  are: 

1.  Validation  exoeri moots.  These  experiments  would 
consist  of  toe  deoumino  of  hardware  or 
software.  The  feasibility  of  the  system  to  work 
accnrdina  to  fixed  specifications  is  determined, 
in  example  would  ne  the  debuaoino  of  the  UPINT 
software  oner  to  o  1  a  c  i  n  a  it  on  line  for 
ooprationa)  testinq  or  use. 

2 .  Demonstration  experiments.  These  experiments 
would  he  sn’-ewhat  more  structured  than  'he 
validation  experiment,  in  that  a  scenario  would 
he  followed.  Tne  term  demonstration  refers  to 
the  tvoe  of  outout  from  the  experiment.  The 
outout  wouli  he  mostly  personal  impressions  in 
the  minds  of  the  users  of  the  system  under 
demonstration. 

3.  Assessment  experiments.  These  are  experiments 
whore  trials  are  conducted  over  a  wide  ranoe  of 


conlitions  wi fh  1 i t t 1 e  control  over  sources  of 
error.  Tne  no el  is  to  obtain  an  idea  of  ho* 
well  the  system  o  e  r  f  o  r  m.  s.  The  outout  is 
subjective  opinions  cf  the  experimenters  and 
subjects.  Their  opinions  cf  how  worthwhile  the 
system  is  and  any  ootent i al  uses  are  recorded  as 
experimental  data. 

a .  t  valuation  exoeriments.  These  are  the  most 
riocrous  tyoe  of  experiments.  Experimental 
conditions  are  carefully  controlled.  ^  number 
of  reel icat ions  are  performed  followed  by  a 
formal  analysis  of  numeric  measurement  data. 

The  assessment  »xneriment  was  chosen  as  the  means  for 
determining  the  utility  of  0  P  1  N  T  .  do  prior  experiments 
ccncemina  the  utility  of  0  P I M  T  have  teen  perfone  i  and 
therefore  no  caseline  currently  exists.  The  results  of  this 
experiment  should  fill  that  oac.  Subjective  impressions  ana 
juddments  of  C3  users  are  what  is  needed  to  helc  determine 
the  usefulness  of  0  P I  N  T  and  that  is  what  this  assessment 
will  do. 

The  major  objective  of  this  experiment  was  to  assess 
tne  utility  of  OP  I  NT  as  a  computerized  decision  aid  in  C3 
applications.  Secondary  objectives  which  helped  in  this 
ooal  included  tne  following: 

1.  To  assess  tne  extent  to  which  DPI  NT  facilitates 
thorouan  and  timely  decision  m  a  kino. 
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2.  To  investigate  th«  user-OpI'ir  interface  and  to 
suggest  re^e^i  al  measures  or  alternate  'lesions 
as  enonori  -jfe. 

T  .  To  survey  user  ooi  nion  on  t^e  ocerat  i  on  am 
ootent i al  usefulness  of  0p!M  in  C3  oecisions. 

The  or  i -nary  objective  of  OP  I  NT  is  to  assist  decision 
na<ers  in  the  sfructurino  am  analysis  of  decision  rronlers 
(imey  et  al.,  1  ^  7  o  ,  c.  4 ) .  One  could  i  m  d  1  v  from  tnis  that 
a  better  decision  cr  orohuct  is  delivered  through  the  use  of 
□PINT.  Tne  question  which  is  now  in mediate  is,  "What  is 
better?"  defter  can  not  be  scenario  dependent  or  olayer 
deoendent.  It  -rust  be  general  in  nature.  0  PI  NT's 
cerfor-nance  nay  be  be  measured  in  terms  of  time  ( tne  time 
from  oercention  of  problem  to  decision).  4  faster  decision 
is  often  times  needed  and  therefore  the  capability  to  do  so 
mav  he  termed  "oood  to  have".  But  it  is  only  "good  to  have" 
if  the  duality  of  the  decision  remains  at  least  as  oood  as 
without  the  aid.  In  order  to  assess  this  feature,  a  method 
for  measuring  the  duality  of  the  decision  is  needed.  4s 
stated  earlier,  the  "correctness"  of  a  decision  is  extremely 
difficult  to  measure  if  af  all.  In  this  experiment  no 
attemot  was  made  to  define  or  measure  this  attribute. 

What  remained  was  the  subjective  judgment  of  trie 
subjects  involved  in  the  experiment.  The  subjects  were 
students  from  the  C 3  curriculum  at  the  Naval  Postgraduate 
School.  C3  students  were  used  since  the  assessment  was 
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perfo’Te'i  of  ; )  P 1 1 1 T  1  s  a  h  i  1  i  t  v  to  help  with  C3  type  Droolers. 
These  students  w  e  r  ?>  all  operational  officers  with  a  large 
amount  of  experience  in  operational  problems  in  the  field. 
Trey  reoresenteo  the  Irnyi  Air  Force.  bavv,  ani  Marines. 
Thev  were  in  t  n  e  i  r  last  (sixth)  Quarter  of  studies  and  had 
oeen  exoos»d  to  numerous  decision  models  ana  aids.  The 
subjects  were  Grouped  into  uai rs  to  form  a  decision  team. 
All  toe  subjects  had  teen  trained  in  the  theory  and 
operation  of  0 0  T  ■’  1 T  aort  rad  received  an  least  tnree  hours  of 
instruction  includina  one  hour  of  hanns-on  time. 

Two  tvoes  of  decision  situations  were  used  -  time 
critical  ana  time  not  critical.  It  was  understood  that  the 
■anne  of  tin?  cos s i o i 1 i t i es  is  infinite  and  there  was  no  way 
to  assess  each.  Therefore,  the  two  extremes  that  could  be 
encountered  xere  us°n. 

Two  scenarios  were  followed  wnich  reaui red  the  subjects, 
actina  as  decision  makers,  to  recommend  a  specific  course 
nf  action.  The  two  scenarios  represented  varyinq  complexi¬ 
ties  which  *  ace  a  C3  decision  maker.  They  were  carefully 
designed  to  avoid  the  allegation  that  they  were  either  0 P I N T 
or  subject  dependent  .  The  first  scenario.  a  Cuoan 
blockade  problem,  was  rather  complex.  It  reaui red  an 
in-deoth  assessment  of  intelligence  estimates  in  order  to 
predict  the  occurrence  of  an  uncertain  event.  In  ad¬ 
dition,  the  evaluation  of  numerous  courses  of  action  was 
reauirej.  The  second  scenario  was  an  air  reconnaissance 
problem.  This  one  was  somewhat  easier  as  the  k?v  u  n  c  e  r  - 
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tain  event  111  already  been  assessed  and  the  decision  maker 
was  only  reouired  to  evaluate  the  courses  of  action  which 
were  avail  sole.  A  cooy  of  each  scenario  is  included  in  Ar>- 
cendi  x  n  . 

The  suoiects  olaved  one  scenario  using  OPIM  and  then 
olaved  the  second  in  a  manual  mode  without  the  use  of  OPI^JT. 
The  schedule  was  varied  so  some  subjects  played  the  d  i  f  - 
ficult  Cuban  crohlem  first  and  some  the  easier  reconnais¬ 
sance  oron'en.  In  addition/  the  use  of  0 P I N T  was  varied 
so  it  was  used  alternately  first  or  second.  Fiaure  2?  ocr- 
travs  a  typical  schedule.  Notice  the  added  dimension  of 
the  time  critical  and  time  not  critical  factors. 

It  was  recognized  the  evaluation  measures  used  in  the 
experiment  would  n0t  be  Quantitatively  measuranle. 
Therefore  care  was  tai<en  to  carefully  select  both  measures 
of  effectiveness  f  v'  0  F  )  ard  measures  of  performance  ( *  0  P ) 
which  sucoorted  the  ooals  of  the  exo^riment. 

ft  measure  of  e f f ec t i vene s s  is  defined  as  those 
evaluation  'tisures  which  indicate  the  contribution  of  0  P I N  T 
to  the  overall  decision  process  (  A  C  C  A  T  Ooerational 
Evaluation  Task,  1^78,  o.  b).  Measures  of  oerformance  are 
defined  as  those  evaluation  measures  which  indicate  a  level 
of  technical  performance  relative  to  the  internal 
functioning  of  G  P I  <\  T  itself.  Soth  are  aoclicable  to 
decision  aid  experiments.  The  M  0  E  '  s  and  '*  0  P  '  s  were  oesianed 
so  as  to: 

l.  veasure  the  contribution  of  OP  I  d  T  to  the 
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TIME  NOT  CRITICAL 


CUBAN 
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RECON 
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CUBAN 

SCENARIO 


RECON 

SCENARIO 


OPINT  MANUAL 


TEAM  l(FIRST) 
TEAM  4  (SECOND) 

TEAM  2  (FIRST) 
TEAM  3 (SECOND) 

TEAM  2  (SECOND) 
TEAM  3 (FIRST) 

TEAM  1 (SECOND) 
TEAM  4 (FIRST) 

TIME  CRITICAL 
OPINT  MANUAL 


TEAM  5 (FIRST) 
TEAM  7  (SECOND) 

TEAM  6 (FIRST) 
TEAM  8  (SECOND) 

TEAM  6  (SECOND) 
TEAM  8  (FIRST) 

TEAM  5 (SECOND) 
TEAM  7 (FIRST) 

FIGURE  22: 


EXPERIMENT  SCHEDULE 


decision  process,  not  tne  decision  outcome 


<? .  Be  objective  so  :ic  not  introduce  bias. 

3.  Pe  simple,  to  achieve  economy  and  ease  of  use. 

<i .  Hermit  determination  of  which  functions  within 
the  decision  orocess  are  aided  and  under  what 
conditions. 

5.  Yield  results  in  t  h  e  exrerimenr  which  are 
expendible  to  real  world  ooerational 
envi  moments. 

6.  Be  operationally  olausible  and  intuitively 

unoerstandahle  by  potential  users  (AC CAT 

Ooerational  Evaluation  Tasx,  197.°,  on.  6-9). 

The  followino  is  a  list  of  tNe  v  0  E  '  s  which  «er»  chosen 
for  use  in  the  exoerimpnt: 

1.  fire  plarsed  in  srrivina  at  the  decision.  This 
includes  the  time  from  perception  of  a  problem 
until  a  decision  is  reached.  In  addition,  the 
time  elsnsed  in  the  actual  use  of  the  aid  will 
be  measured. 

2 .  TdenriHcation/elimination  of  non-essential 
information. 

3.  Objectivity  in  leterminino  relationships  among 
factors  contribution  to  the  selection  pf  an 
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alternative  course  ot  action. 


y.  Pis*  oerceiveo  to  oe  associates  w i t n  the 
-.I  e  c  i  s  i  o  n  . 


5.  Ease  of  use  of  the  decision  aid. 


b.  Ease  of  modification  of  data  within  the  model. 


7 .  Perceived  advantaoes/disadvantaaes  aained  from 
the  use  of  the  decision  aid. 


Ease  of  recovery  from  user  errors. 


Oata  «as  oathereo  on  each  of  the  above*  tnrouah  the  use 
ot  ouesticinoai  res.  The  exception  was  the  first  ^OE .  For 
this  one*  times  were  recorded  while  the  subjects  were  using 


0 P  I  N< T  to  caoturp 

the  amount 

of  time  required 

t  0 

build 

the 

mode  1 /  time  to 

evaluate 

the  data/  and 

total 

time 

t  o 

dec ision.  These 

times  were 

then  compared 

with 

times 

t  o 

evaluate  data  and  total  time  for  the  manual  operation. 
Three  questionnaires  were  used  to  record  the  opinions  of  the 
subjects.  A  questionnaire  was  comoleted  after  the  ol av  of 
each  scenario  and  one  completed  after  both.  Sample 
auest i onna i res  are  included  as  Appendix  C. 

Ourino  the  actual  play  of  the  scenarios/  a 
cons i derat i on  was  how  to  satisfactorily  place  the  subjects 
into  the  mental  role  required.  This  includeo  the  ability  of 
the  subjects  to  adeauately  assess  orobab i 1 i t  i  es  of  future 
events  and  the  reqrets  required  in  the  construction  of  the 
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model.  This  research  design  subsumes  this  cons i derat i on  by 
locatinu  crodabilifv  estimates  within  the  scenarios  and 
aef ininj  the  aoal s  and  ohiectives  to  be  reached.  Subjects 
were  encouraaei  to  nrowid®  their  own  rearet  assessments  it 
they  had  e<oerience  in  similar  problems.  If  the  suhjpcts 
had  no  previous  experience#  an  individual  was  available  to 
act  as  a  senior  official.  The  senior  official's  role  was  to 
orovide  renret  data#  if  needed#  and  to  answer  Questions 
concernin':  uncertainties  or  amniouities. 


v  I  .  T  H  F. 


RESULTS  OF  [ME  EXPERIMENT 

Tnree  s i i f i c t  results  f ror  the  experiment  were 
revealed.  First/  nearlv  all  (85%)  subjects  stated  they 
liked  the  caoabilities  which  0  P I N  T  nave  them  and  would  use 
U  p  I '!  T  in  the  future  if  it  was  available  to  them.  Second/ 
a  1  t  *  o  u  a  h  tney  1  i  k  e  d  the  caoabilities  o  *  0  P  I  i'j  T  ,  all  (100%) 
stated  thev  did  not  like  it  in  its  present  form/ 
oarticularlv  in  tne  ,rea  of  user-interface.  Finally/  a 
oroarammino  t?ua  (error)  was  discovered  in  the  use  of  the 
sensi t i vi tv  analysis.  The  sensitivity  analysis  Permitted 
the  comparison  of  only  trre»  courses  of  action/  showinp  the 
expected  regrets  for  the  orders  as  zeros. 

It  is  important  to  keen  each  of  these  results  in  mind 
as  1 1»  e  sceci#ic  results  are  read.  wany  negative  comments 
were  conditioned  ov  the  statement  that  if  user-interface  was 
better/  or  if  t  te  sensitivity  worked  correctly/  the  comments 
would  be  cositive.  The  cases  where  this  occurs  are 
aoo r or r i a t e 1 y  noted. 

The  sensitivity  analysis  oronrammino  "bug"  was  found 
durina  the  olay  of  the  Air  Reconna i ssance  Scenario.  This 
scenario  had  five  possible  courses  of  action  which  needed 
evaluation.  OPI’hT  handled  the  exoected  value  and  comoined 
value  calculations  perfectly.  But  whenever  a  sensitivity 
analysis  was  asked  for,  OPINT  would  not  comoute  the  expected 
regret  for  all  courses  of  action.  Additional  research 
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revealed  that  in  the  case  of  a  sensitivity  analysis  nhere 


the  ocnfiaoi  1  i  tv  of  the  event  was  varied/  0  P  I  N  r  w  o  u  1  d  only 
compute  three  courses  of  action.  .hen  the  variation  of  the 
criterion  weights  was  as<eh  f  o  r ,  0  P  I N  T  would  compute  four 

courses  of  action.  Any  rpraiiina  courses  of  action  received 
rearet  values  of  zero.  This  did  not  Hermit  a  complete 
evaluation  of  alternatives  and  had  an  adverse  effect  on  the 
f e e l i m s  of  the  subjects  toward  the  decision  aid.  Although 
0 P I N T  had  only  c®en  installed  on  the  POP  11/70  for  one  month 
prior  to  the  experiment,  oroorammino  bugs  such  as  these 
should  have  been  detected  and  corrected  prior  to  presenting 
the  aid  as  operationally  ready. 

All  but  four  subjects  said  they  felt  the  scenarios  were 
realistic.  These  four  felt  that  priorities  established  in 
the  scenario  were  not  what  they  felt  they  should  be  and  that 
orior  decisions  made  did  not  seem  realistic  from  their 
viewpoint.  The  use  of  the  realistic  scenarios  coupled  with 
the  ability  of  the  suoiects  to  input  their  own  probabilities 
and  regrets  facilitated  a  decision  maxing  environment  that 
was  as  realistic  as  possible. 

The  eight  M0F  *  s  stated  in  Chapter  Five  were  evaluated 
through  t  n  e  use  of  hupstionnai res.  A  summary  of  the  results 
for  each  is  s  n  o  w  n  below: 

1.  Time  e' arsed  in  arriving  at  the  decision.  Times 
were  recorded  in  order  to  capture  the  time 
subjects  were  spending  on  the  scenarios,  both 
with  and  without  the  utilization  of  0  P I M  T . 
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Those  t i "ss  w°re  recorded  int  to  determine  if  a 
faster  decision  was  beinq  maoe»  hut  to  provide 
an  indication  of  ho*  1  o  n  q  it  took  to  build  the 
DPI  -  i  T  model  ano  evaluate  it.  In  all  casesi  more 
time  was  soent  while  using  0  P I N  T  than  in  the 
manual  state.  The  time  renuireci  to  build  the 
UP  T  N  T  mode!  avera-ned  A  5  minutes  for  the  Cuban 
scenario  and  P'S  minutes  for  the  Air 
P ec onn a i s s a nc e  scenario.  Toe  times  were  tiqhtlv 
Grouped  as  is  shown  by  standard  deviations  of  4 
and  £  minutes  resoec t i ve 1 v .  The  time  to 
evaluate  THE  0 p I N T  model  averaqed  ?  1  minutes  and 
iu  minutes.  Aqain  small  standard  deviations  of 
j  and  1  minute  sho«  tiant'y  aroucea  data.  These 
times  are  very  near  to  what  was  expected  and 
snow  a  General  ability  of  the  subjects  to  build 
ano  manipulate  the  model. 

Identification/el  in ination  of  non-essential 
information.  The  qoal  here  was  to  determine  if 
the  decision  orocess  was  aided  by  identifying 
non-essential  information.  6  3  £  of  the  users  of 
OP  I  NT  were  able  to  identify  some  non-essential 
information  while  only  31  were  able  to  in  the 
manual  mode.  The  items  of  information 
identified  varied  accordinq  to  subject-specified 
probabilities  and  reorets.  An  additional  a  q  %  of 
the  0  P I \ T  users  stated  problems  with  the 
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r  i  vi  t  y  analysis  cr?venten  the1"  from 
rossirly  i  fyjnn  any  non-essential 

i n  f  or ma tier. 

3.  Or i ec t i v i t  y  in  determinino  re ! at i onsN ics  a  m  o  n  3 
factors  contribution  to  the  selection  of  a 
course  of  action.  The  ooal  of  this  'HIE  was  to 
determine  if  the  subjects  were  aided  in  the 
decision  omcess  ry  de  t  e  r  t  i  n  a  t  i  c  n  of  which 

factors  were  the  most  critical.  b3%  felt  U  P I \ 1 
definitely  aided  in  tne  decision  orocess.  These 
subjects  definitely  liked  the  way  OPJ'iT  promoted 
detailed  and  non-subiective  analysis  of  what  was 
aoina  to  haonen  and  why.  The  rest  found  it 
ccnfusino  due  to  uncertainty  of  what  the 

disolavs  were  actually  t e 1 1 i n a  them.  6  32  also 
stated  they  were  able  to  assess  the  criticality 
of  the  contributino  factors  when  us  i  nq  OPI'iT. 
The  evaluator’s  observation  here  was  that  in  the 
manual  mode,  one  factor  was  determined  to  be 
critical  and  then  the  rest  ignored.  This  din 
not  occur  when  0  P I  d  T  was  used. 


4.  Pis*  perceived  to  be  associated  with  the 
decision.  This  v  0  £  *  a  s  used  to  measure  the 

deoree  of  uncertainty  or  certainty  in  the  final 
decision.  BSX  of  the  subjects  in  the  manual 

mode  said  they  made  the  best  decision  possible 
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while  only  5  ->  %  in  t  n  e  0  P  I N  T  ,no<ie  f»]  t  mat 
certain,  iwc  tynas  o*  comments  were  n ■  a  d  e  . 
First1/  Me  sensitivity  opopI  p"  uas  aaaio 
mentioned  as  a  him  ranee.  Seconoi  several 
suh  iects  Felt  uneasy  about  what  the  numbers  in 
the  displays  actually  meant.  For  example/  in 
the  expected  value/  combinec  value/  and 
sensitivity  ni sol 3/S/  only  numbers  are  shown. 
There  is  no  explanation  of  what  they  mean  or 
which  numbers  are  best. 

Ease  of  use  of  the  decision  aid.  This  MDt  was 
used  to  heir  determine  what  type  of  user* 0 PINT 
interface  existed.  Only  31%  stated  that  OP  I  NT 
presented  displays  in  a  manner  «hich  was  easily 
understood.  Sensitivity  and  combined  value 
displays  received  the  most  comments  as  beino  the 
most  difficult  to  understand.  Also/  there  were 
several  comments  concprni na  the  inputs  reauired. 
'•’any  were  not  sure  what  was  actually  being  asked 
for  and  when  to  hit  the  carriage  return  and  when 
not  to.  Almost  all  B5%  stated  the  displays  were 
generated  quickly  emuah.  No  problems  were 
observed  hero,  even  'luring  times  of  ceak  usaoe. 
Additional  comments  concerned  the  use  of  menus, 
'-’one  efficient  movement  throuon  the  menu  levels 


was  wanted 


Ease  of  modification  of  -fata  within  the  model. 
Toe  goal  oere  ^as  to  letaroi  how  easv  i  r  was 
to  correc'  » h  e  t.  any  errors  that  untrained: 
tynists  will  make.  8  S  %  state'  it  was  easy  to 
ma*e  corrections  in  the  model.  The  only 
neoative  comments  concerned  the  use  of  the  menus 
for  tois  purpose.  There  existed  too  much 
refracino  throunh  menu  selections  to  make 
corrections  in  toe  same  area  or  level  of  the 
model  . 


Ease  of  recovery  from  user  errors.  This  ,Ji  0  E 
measure?  now  well  the  G  P I T  orooram  recovered 
from  input  errors.  here  the  goal  was  to 
net erm i n»  if  nor»al  typing  or  careless  errors 
caused  major  crcPlems.  QP  Z  o f  the  suojects  who 
maoe  errors  felt  recovery  from  these  errors  was 
poor.  Comments  centered  around  some  simple 
mista*»3  which  caused  major  oronlems.  during 
initial  disolaySf  a  selection  of  a  choice  is 
made  ov  moving  the  cursor  to  the  desired  choice 
ano  then  h i t t i nc  the  'X'  key  for  execution, 
however  in  six  cases#  subjects  pressed  the  'X' 
key  followed  ny  a  carriage  return.  This  caused 
tho  orooram  to  halt  ana  the  subjects  were 
required  to  oeoin  again.  Several  subjects  felt 
that  neculous  input  directions  caused  several 
on-pr 3  w  n i c  h  coulo  not  be  changed  until  after 
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t  n  e  ?nt  i  re  "i  o  i  e  1  was  co»olet?J. 

pnrc°i  vo'!  I'ivarfenes/nisa'lvsr'taqes  gained  f  ra"! 
t  re  use  c  f  r  P  I  '"j  T  .  The  “solicit  consideration  of 
all  factors  tirouah  the  promotion  of  a  good 
rhouqht  orocess  was  the  comment  most  often 
stated.  Several  stated  they  liked  beina  forced 
into  a  looical  thouoht  orocess,  Also#  the 
subjects  felt  that  OpIi"T  orovided  them  with 
locurent  af ion  of  their  decision  and  would  make 
it  easier  to  present  the  outcome  ana  justify  tne 
conclusion.  The  d i s a a v an t aoe s  listed  for  using 
DP  I  \|T  as  it  currently  exists  centered  around  the 
user-interface  o  r  o  n  1  e  m  .  There  e*i  st°n  a  general 
lack  of  comfort  due  to  the  limited  descriDtions 
and  assistance  from  the  aid. 

An  additional  guest  ion  concernina  training  was 
asked  of  each  subject.  They  were  asked  to 
comment  on  the  amount  of  training  they  felt  was 
reauired  for  0  P I m  T  usage.  The  overwhelming 
resoonse  was  about  two  hours  of  structured 
instruction  on  r ' ,  e  theory  and  use  of  the  aid# 
follower:  by  one  to  two  hours  of  hands-on 
demonstrations.  This  amount  of  instruction  was 
exactly  what  each  subject  received  crior  to  the 


BP 

. -  — -  - — 


e*oeri  oienf 


v  L I .  CT.-CUiStr.fiS  Af!r>  ^Frn'-'vrf-QAT  ruf'js 


It  was  aetorTined  that  0  P I  d  I  is  b  e  n  e  f  i  c  i  a  1  to  the 
decision  m  a  *  e  r  in  t  h  p  decision  m  a  k  i  n  q  orocess.  The  majority 
o f  the  subjects  cOTKpnts  were  either  oositive  in  relation  to 
OaIuT  or  „  o  u  1  d  "iav°  been  with  either  no  prooran  errors  or 
petter  user-interface.  P  P  I T  1  s  oreatesf  advantane  lies  in 
two  areas.  First,  if  promotes  ( a n d  possibly  even  forces) 
toe  user  into  a  scuna  tto'jatf  process.  In  toe  neat  of 
crisis  or  even  just  day-fo-day  decisions,  it  is  o  f  t  °  n  too 
easy  to  foroet  the  principles  of  need  decision  m  a  k  i  n  o  and 
slip  to  semetnino  1  #><15 ,  Tkq  use  of  OFI-iT  could  heir  to  keen 
tnat  from  haonenina.  Second,  OF  T  J  T  provides  its  own 
documentation.  This  documentation  is  t  n  e  n  available  to 
present  decisions  and  snow  w w  v  a  Particular  decision  was 


made 

It 

i  s 

also 

easy  to  focus  on  the 

critical  areas 

a  n  1 

not 

waste 

time 

0 n  fro 

non-importanf  ones. 

There 

are 

still 

sort  "a jor  flaws 

whicn  need  to 

be 

worked  out  of  the  svstem.  The  first  and  foremost  is  the 
sensitivity  analysis.  Proaram  buns  such  as  these  can  not 
e  *  i  s  t  in  an  aid  if  if  is  to  be  useo  crooerlv.  becond,  is 
the  user-interface  Problem.  The  aid  must  ho  oe sinned  for 
the  casual  user,  on?  who  at  one  t i m e  may  have  been 
proficient,  out  throumh  lack  of  use  has  fernotten  the  key 
re-juirements.  Additional  comments  conc°rnim  wnat  is  needed 
or  what  is  bains  displayed  are  a  must. 
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T  n  o  TQVemenf 


+  t  "  e  user  the  a  i  a  via  the 


menus 


also  neeis  attpnrrn,  users  s h o u  1  a  be  liven  the  choice  of 
returning  t- o  se«»ril  levels  rsfner  tnen  onlv  uo  ore  or  down 
nop  . 

T"s  final  ano  fcy  no  means  the  least  inoortant  is  the 


oroclem  of  orooram 

halts.  'Mcthing  ori  eves  a  user 

more 

than 

havina  to  start  ov»r#  especially 

after  a 

oreat  deal 

o  f 

work 

has  already  been 

done.  The 

S  V  S  t  em 

current  1 y 

d i sol avs 

numeric  incuts  for 

veri  f i c  a  t i 0  n  . 

T  n  e  r  e 

i s  no  reason 

why 

a'chanetic  or  menu 

choices  could 

not  also 

re  none . 

T-is  autnor  n.js  several  recommendations  for  both 
i  morov  i  ">0  ti=  decision  aid  and  for  future  exoeriments  with 
if.  -s  stated  earlier,  the  a  i  d  needs  to  ce  desioneo  for  tne 
notional  user.  This  user's  knowleriae  of  the  aid  could  r  a  n  q  e 
from  err  1 et  e  fo  nothino  lecendi no  on  amount  of  usaoe.  The 
aid  snoulci  ce  d^s  i  one  1  *  or  tne  two  extremes  and  even  for  the 


user  in  tn& 

mi  idle. 

It  neons 

to  re 

both 

user  friendly"  and 

"error  t  o 1 

e  r  a  n  t  "  . 

0  n  e 

t  ec hn  i 

due 

could  be  optional 

v  e  r »  osi  ty. 

for  each 

display* 

three 

sets 

of  comments  could 

be  n reoared.  The  decision  maker  then  selects  which  set  is 
desired  ascending  on  need.  The  knowledqeaole  user  would 
roceiv»  only  short  Phrases  which  could  be  keyed  on  to 
de"e rate  in ruts  or  disc  lav  results.  The  same  tvoe  of 


c  3oab i 

1  i t  v  wou 

1  ^  ex 

i  s  t 

*  or 

t  hp 

other 

tyres 

of  users.  This 

wav  t 

■>  p  user 

rpc^i 

v  e  s 

o  n  1 

1  V 

what  i 

3  neeced . 

Anot  her 

v  a  r  i  a  t 

i  in  Of 

t  ^  i  s  i 

s  t  o 

allow 

each  user  to 

so»c  i 

f  v  exactly 

what 

comment  3 

e  r  o 

r>  a  a  r  e  'i 

t  0 

i  n  s  ij  r  p 

correct 

u  s  a  a  e  and 

i  n  r  e  r  o  r  -»  t  a  r  ion 


T  h  o 

\  t 

ion  o  +  SOme  sort 

of  " 

HELP  " 

command  would 

also 

h  e» 

u  s  e  f 

u  i  • 

It  a  1»c ' si  on 

m  a  x  e  r 

nets 

stuck  and  neens 

some 

oroTct in 

hn  /■>  p 

inrernretatiod/ 

a  a  i 

mp  1  e 

r  ecu®  s  t  woulu 

fill 

that 

need. 

Usnr  confidence 

could 

improve  areat 1 v  with 

this 

s  i  to  le  ani  t  i  on. 

Once  0 13 1  N  T  is  rortesi  nred  to  meet  the  neeos  of  the 
users/  as  state  i  in  rh»  results/  additional  experiments 
snould  oe  conducted  fo  measure  additional  caoaoi 1 i t ies  of 
the  aid.  This  ^*ceri-"onr  din  not  consider  whether  a  better 
decision  was  he  inn  made.  This  is  net  an  easy  problem  and 
further  research  and  experimentation  is  called  for. 

Anotror  area  for  e*o»rin»ntat ion  has  to  do  with  the 
area  of  rearet  values.  0 p  I  d  T  currently  is  desicned  to 
handle  r^orpts  rather  t.  nan  gains .  Some  subjects  casually 
stated  they  minor  have  preferred  to  use  oains  rather  than 
reorets.  Additional  experiments  to  determine  if  the  use  of 
oains  vice  regrets  has  an  i m o  a  c  t  on  user  ef  f i c i enev  is 
definitely  called  for. 

Finally/  thouoht  should  he  niven  to  providing  a  hard 
cooy  notion  to  us°rs  who  have  only  a  soft  copy  (CRT) 
caoahility.  Further  experimentation  could  be  done  to 
determine  if  potential  users  want  or  ev®n  need  a  hard  cooy 
carahi  1  i t  y  . 

The  underlying  premise  of  these  results  is  t  nat  most  of 
the  rouble  encountered  in  aettinoi  computers  used  in 
decision  Taxing  comes  from  too  m  u  c  h  emphasis  on  bits  and 

8F, 


bvtes  ann  -nini^^l  attention  on  the  user.  The  use  of 
decision  suocorf  systems  s  u  c  h  as  t  •  P I  “  i  T  r  o  f  f  ®  r  an  oonortunitv 
t-nat  is  foth  trivial  ana  ifnense.  If  is  immense  in  that 
they  can  rh»  comruter  in  decision  n-aicina  activities 
where  large  oayotfs  could  be  o  r  o  v  i  d  e  d  anri  trivial  in  that 
trey  reorespnt  no  major  advance  in  technology.  Ruildina 
innovative  systems  will  be  a  difficult  am  risify  venture  for 
some  fine  to  cone.  The  future  of  decision  suoocrt  systems 
ana  coerational  Tec ’ si  on  aids  d  e  o  e  no  s  on  some  adjustments 
fhat  both  the  developers  aog  users  must  make.  But  these 
adjustments  are  small  ones*  and  are  mainly  attitudinal. 
Little  new  knowledge  is  reouired.  but  the  cooortunities  are 
all  substantial*  and  t K  e  oersonai  and  organizational  rewards 
High. 


B6 


APPENDIX  A 


SAMPLE  QPINT  CESSION 


oni  nt 

1)  Ann  Arbor 
? )  Anv  hardcoov 
3 )  fl'v.i  3  a 

■<-  h  i  c  h  terminal  t  y  o  e  are  vou  u  s  i  n  a  ?  2 

Please  select  one  of  the  followina  oof  ions. 

OPTION 

1  Di ?ol av  results 

2  Edit  no net 

3  Sensitivity 
a  Leal  rolel 

S  Create  of  a  in  to  model 
b  Save  mode  1 

7  New  information 

8  End  session 
Enter  desired  ootion:5 

Is  this  a  new  m  o  o  e  1  ?  (  v  or  n):y 

Please  enter  a  title  for  this  model;  lair  reconnaissance  1 

A'hich  nortion  of  the  model  would  you  like  to  create? 

OPT  r ON 

1  Generate  influence  liao  ram 
?  Generate  value/rearet  model 

3  Aod  Bavsian  indicators 

4  Pet  urn  to  main  menu 
Enter  desired  ootion:l 

.'.hat  is  the  mein  event  of  interest?  I  k  o  r  e  a  n  intent  ) 
korean  intent 


i  i 

1  I-  t  1 

Please  enter  the  outcome  lacels  for  the  event  you  have 
called  korean  intent 

Enter  outcome  label:  (no  action  1 
Enter  outcome  label:  [harassment! 

Enter  outcome  label:  [shoot  sown) 

Enter  outcome  label:  [  1 
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Please  enter  the  oroh^oi 1 t  v  of  each  outcome  of  the 
event  vou  nave  1  a  n  e  I  e  t  k  o  r  e  a  n  intent 


f 


no  action  harassment  shoot  down 

-bO  30  tu -  - 

'lcr”al  izeo  values:  oO.O  3  0.0  10.0 

If  these  values  are  correct  tyoe  ao:  [qol 

A'hich  oort  i  on  of  the  model  would  vou  H<e  to  create? 

OPT  I  OH 

1  Generate  influence  diaqram 

2  Generate  value/reoret  model 

3  Add  Bavsian  indicators 

il  Pef  urn  to  main  menu 

Enter  desired  oot ion:? 

You  will  Be  asked  to  ornvioe  t  h  e  criterion  to  be  used 
in  your  value  junuement,  the  actions  you  may  take,  ann 
the  associated  value/reoret  matrices  for  each  criterion. 
Please  press  return  to  continue. 

Enter  the  names  for  eacn  of  your  criterion; 


Criterion:  ( a i r c r a f t - a i rc r e.  ) 
Criterion;  [loss  Of  information  ] 
Criterion:  [national  influence  1 
Criterion;  [recon  oroaran  1 
Criterion;  [  ) 


Now  enfer  the  labels  for  each 
Action:  (do  not  fly  J 

Action;  [modified  route  1 
Action:  (armed  escort  1 
Action:  [hiah  cerf  a/c  1 
Action:  (normal  mission  1 
Action:!  ] 


action. 


You  will  now  be  asked  to  provide  the  values  for 
each  of  these  value  matrices. 

Please  oress  return  to  continue. 

Please  enter  reqret/value  matrix  for 
ai rcraft-ai rcrew 

no  action  harassment  shoot  down 


do 

not  f  1 

V 

0 

O 

o 

1 

1 

do 

not  f  1 

y 

0.0 

0 . 0 

0 . 0 

I  f 

these 

values 

are 

correct 

tvoe  oo: 

!qo! 

8« 


modified  rout1? 

0 

-  2  0  -o  0  - 

— 

-  - -- 

itol  i  f  i  eu  route 

0.0 

-20.0 

-#>0 . 0 

It  t  n  e  s  e  values 

are 

correct1 

t  voe  no  : 

loci 

armed  escort 

0 

-IS  -35- 

-  - - 

--  -  . 

ariei  escort 

0  .  o 

-  1  S  .  0 

-35.0 

It  these  values 

are 

C  o  r  r ec  t 

t  voe  qo  t 

(qo  1 

hicih  o erf  a/c 

0 

-S  -IS  - 

.... 

.  .  .  . 

h  i  ah  oerf  a/c 

0.0 

a 

• 

o 

-  1  s .  0 

It  these  values 

are 

C  O  r  r ec  t 

t  voe  qo : 

(oo) 

normal  mission 

0 

-25  -100 

.... 

.... 

ncrual  miss i co 

o.o 

-25.0 

- 

1  0  0  .  o 

It  these  values 

are 

c  o  r  r  ec  t 

t  voe  go : 

f  oo  1 

PI eas^ 

enter  rearat/value  matrix  for 

c  u  r  r 

ent  information 

no  action  h a r a s smen t s hoo t  down 

»*- 

o 

C 

C 

r> 

V 

-10  0  -100  -10  0 - 

rlo  not  f  1 

V 

• 

o 

o 

• 

o 

• 

o 

o 

1 

-10U.0 

It  these 

va 1 ues 

are  correct  tyne  go: 

1  no  1 

mo  1 i ties 

route 

-30  -  50  -30 - 

m o hi  t  j  e <1 

route 

-30.0  -50.0 

-  3  o  .  0 

It  these 

va 1 ues 

are  correct  t voe  go: 

loo) 

armed  escort 

-S  -S  -5  - 

armed  escort 

-s.o  -s.o 

-S.O 

It  these 

v  a  1 ues 

are  correct  tvpe  go: 

l  qol 

h inn  oe  r  t 

a/c 

-SO  -SO  -SO - 

h i qh  oe r  f 

a/c 

-50.0  -50.0 

-50.0 

It  these 

values 

are  correct  tyne  oo: 

too  1 

no  r ma 1  mi 

ss  i  oo 

0  0  -100  - 

norma  1  m i s  s i on 

0.0  0.0 

-100.0 

It  these 

v  a  1 ues 

are  correct  fyoe  oo: 

1  qo] 

PI  ease 

enter  regret/value  matrix  for 

national  influence 

no  action  harassment  shoot  down 

do  not  1 1 

V 

-SO  -100  -70 - 

do  not  1 1 

V 

-S0.0  -100.0 

-70.0 

It  these 

v  a  1 ues 

are  correct  tyoe  ao : 

[qo  1 

mod i f i ed 

route 

-20  -20  -SO - 

•modified 

route 

-20.0  -20.0 

-50.0 

It  these 

val ues 

are  correct  tvoe  oo: 

l  ao ) 

80 


a  r  med  escort 
armed  escort 
[ *  these  values 


- a  0 . 0  0.0 

are  correct  f  yrf>  no:  too] 


0  .  0 


h i ah  n« r  f  a/ c 
h  i  oh  cerf  a/c 
If  these  values 


-25  -10  -10 - 

-2*5.0  -10.0 

are  correct  tyoe  ao: 


-10.0 

tool 


nopral  rj  ssion 
normal  mission 
If  these  values 


0  -25  -100 - 

0.0  -25.0 

are  correct  tvoe  oot 


-10  0.0 

tqo  1 


PI  ease  ent er 

r ec nnn a i s s arc e  oroo 

no  action  h a r a 3 smen t s h oo t 


rio  not  fly 
do  not  fly 


-10  0  -100  -10  0 - 

-100.0  -100.0 


ue  matrix  for 
down 
10  0.0 


If  these  values  are  correct  tvoe  oo:  fool 


modified  route 
modified  route 


-70  -50  -50 - 

-70.0  -50.0 


If  these  values  are  correct  tvoe  oo:  tqol 


armed  escort 
armed  escort 


n  o  — 
0.0 


0.0 


■50.0 


0 . 0 


If  these  values  are  correct  tyoe  do:  tool 


hinh  oerf  a/c 
hiqh  oerf  a/c 


-10  -10  -10 . . 

-10.0  -10.0  -10.0 


If  these  values  are  correct  tyoe  ao:  fool 


normal  mission 
normal  mission 


0  -70  -80 - 

-10.0  -70.0 


•80.0 


If  these  values  are  correct  tyoe  no:  too) 


Please  enter  a  set  of  importance  weights  for  these  criterion. 

ai rcraft-ai rcrew 

current  information 

national  influence 

reconnaissance  oroq 

wts:  100  10  50  30 - - 

Norm:  58.8  5.0  17.6  17.6 

If  these  values  are  correct  tyoe  oo:  Iqol 


Anich  oortion  of  the  monel  would  you  like  to  create? 
OPT  ION 

1  Generate  influence  diaaram 

2  Generate  value/rearet  model 

3  Add  Bavsian  indicators 

4  Return  to  main  menu 
Enter  desired  oof  ion:  4 


<50 


Please  select  one  of  the  following  ontions. 

opt  I  or;  '  " 

1  Pis n lev  result^ 

2  Edit  no  ie  1 

3  Sensitivity 

•l  L  o  a  d  model 

5  Create  or  adg  to  model 
b  Save  mode  1 

7  Mew  infor-nat  ion 

S  Eno  session 

Enter  desired  notion:! 

The  following  disolays  are  available 

OPT  ION 

1  Exnecteo  value 

2  Conoined  value 

S  Event  likelihood 

■*  Values 

5  V  a  1  u  e  w  e  i  o  h.  t  s 

h  Influence  d i aarar 

7  Return  to  main  menu 

Enter  desired  opt  ion:  1 

Expected  value/regret 


do  not  f 1 y  -35.4 
modified  '•/-juf-o  -23. p 
armed  escort  -p.2 
h i ah  cer f  a/c  -9.P 
normal  mission  -20 . 1 


Please  cress  return  to  continue. 

The  f o 1  I o w i nq  disolavs  are  available 
OPTION 

1  Expected  value 

2  Comoined  value 

3  Event  likelihood 

9  Values 

5  Value  weights 

b  Influence  diagram 

7  Return  to  main  menu 

Enter  desirel  ootion:2 


Comb i ned  Value 


no  action  h a r a s smen t s h no t 


do  not  f  1  y  -32.  ij  -u  1 . 2 
modified  route  -17.6  -2S.P 
armed  escort  -7.4  -9.1 
highoerfa/’c  -9.1  -9.4 
norma!  mission  -1.3  -31. S 


Pleas?  press  return  to  continue. 


down 

-35.9 

-54.7 

-20.9 

-15.3 

-96.5 
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The  folio  *  in  a  d  1  s  o  1  a  v  s  are  a  v  a i 1  a  o 1 e 

;)P  r  in': 

1  Expected  value 

2  Corhi  nel  value 

3  Event  1  i  <  e 1  i h ood 

0  V  a  1  u  e  s 

5  Value  w  e  i  a  h  t  s 

o  Influence  d i a o r a m 

7  Return  to  main  menu 

Enter  desired  ootion:7 

please  select  one  of  the  followinq  ootions. 

OPT  I OM 

1  Disol av  results 

2  Edit  mooel 

3  Sensitivity 

-l  Load  model 

5  Create  or  add  to  model 

o  Save  model 

7  New  information 

8  End  session 
Enter  desired  ootion:  5 

Select  one  of  toe  followinq: 

OPTION 

1  Event  probability  thresholds 

2  value  weioht  thresholds 

3  Return  to  main  menu 
Enter  desired  notion:! 

,'l  h  i  c  h  event  outcome  livelihood  should  he  varied? 
OPTION 

1  no  action 

?  harassment 

3  shoot  down 

Enter  desired  ootion:! 


Expected  Value  when 
Probability  of  no  action  is: 


0  10 

20  30  00 

50 

6  0 

70 

80 

q0  100 

do  not  fly  -39  -38 

-37  -37  -3o 

-35 

-39 

-39 

-33 

-32  -31 

modified  route  -32  -31 

-29  -27  -26 

-2  a 

-23 

-21 

-20 

-18  -17 

armed  escort  -ll  -11 

— 10*  —  to*  -o* 

-9* 

-8* 

-8* 

-7* 

-7 

-6 

hi  oh  oerf  a  /  c  -  l  0  *  -  1  0  * 

-1  Q  *- 1 0  *- 1  0 

-10 

-10 

-10 

-Q 

-9 

-9 

normal  mission  -02  -38 

-39  -30  -2b 

-22 

-18 

-10 

-10 

-b  *  -2* 

Please  Dress  return  to  continue. 


Select  one  of  tne  followina: 

OPTION 

1  Event  orobahilitv  thresholds 

2  Value  weioht  thresholds 


02 


5  Return  to  main 

Enter  desired  ort  ion:  5 


m  a  n  y 


w  1  ease 
ot-'T!nf; 
1 

<? 

3 

4 

5 

6 

7 

8 

E  n  t  e  r 


splect  one  of  the 

r  i  5  O  1  -3  y  results 
E  d  i  t  Tone* 
Sensitivity 
Load  model 
Create  or  add  to 
Save  mode) 
dew  information 
End  sess i on 
desired  not ion:6 


f  o 1 1 o  w i na 


on  t ions. 


node! 


p!ease  select  on®  0f  tie 

OPT  10', 


1  Oisolav  results 

2  Edit  monel 

S  Sensitivity 

9  Load  model 

^  Create  or  ado  to 

o  Save  model 

7  ‘'Jew  information 

M  End  session 

Enter  desired  ODfion;f 


foil Ow l nq 


mode  1 


Cot  ions. 


iiPPFl.O  l  x  B  -  SAMPLE  SCENARIOS 

CUBAN  BLOCKADE  S C E 'j A f?  1 G 

On  E5  vt  arch  1  Q  « 0  *  ohotcdranhs  received  from  military 
surveillance  aircraft  showed  the  existence  of  new 
construction  along  the  southern  coast  of  C  u  c  a  .  Intelligence 
analysts  believe  t  h  »  construction  to  be  a  new  major  Soviet 
nuclear  suf'eMre  suooort  rase.  however*  they  are  u’deci  lea 
as  to  w  n  e  t  h  e  r  it  will  he  caoable  of  resurolyini  only  nuclear 
fuel  or  ooth  nuclear  fuel  and  nuclear  S  L  d  v  missiles. 

Additional  intelligence  from  other  sources  has  revealed 
the  presence  of  hunjreds  of  new  technicians  in  the  area.  In 
response  to  American  inquiries*  the  Soviets  have  stated  that 
(■hoy  are  merely  n«l  oi  "a  the  Cubans  ouild  a  new  merchant 
Dort.  They  claim  the  technicians  will  leave  as  soon  as  the 
oort  is  comoleteo. 

u  o  s  t  US  ooservers  are  hianly  concerned  over  these  new 
developments.  Trey  f »  e  1  the  oresense  of  a  new  port*  caoaole 
of  nuclear  retrofit  for  Soviet  submarines*  has  a  serious 
imoact  on  US  national  security.  'lo  lonaer  would  the  Soviet 
submarines  be  required  to  return  to  home  ports.  They  could 
conceivably  oatrol  near  our  shores  for  unlimited  periods  of 
time.  If  a  nuclear  v  a  r  ever  did  b  r  e  a  *  but*  a  missile 
resuooly  base  so  near-hy  cnuld  be  devastating. 

Analysts  relieve  t  h  <=  new  construction  is  caused  bv  the 
oossibilitv  of  non-rat i  f test  ion  of  the  new  SALT  Treaty. 


Tnev  feel  if  the  Treaty  is  not  ratified 


the 


Soviets 


will 


have  a  jump  in  arms  escalation  and  the  US  will  have  no  cower 
to  interfere.  The  analysts  also  oelieve  that  if  the  Treaty 
is  ratified/  tne  construction  will  quickly  cease. 

I'n  the  morning  of  2  7  M  a  r  c  h  ,  three  Soviet  shios  were 
detected  sailing  towards  Cuba.  Reconnaissance  has  revealed 
one  to  be  a  freighter#  one  an  arired  escort  destroyer/  and 
tne  last  an  escort  submarine.  Intelligence  sources  in  the 
Soviet  Union  have  said  the  freiqhter  is  carryina  the  first 
suorly  of  nuclear  fuel  and  may  oossiblv  be  also  carrying 
SLav,s. 

Fears  that  the  Soviets  may  oe  trying  to  auickly 
establish  a  stronghold  in  Cuba  oromntea  the  President  to 
order  a  tnree  ship  Task  Force  to  intercept  tne  Soviet 
freighter.  Fhe  mission  of  the  Task  Force  is  to  determine  if 
nuclear  suoolies  of  any  type  are  on  boaro/  ana  if  so/  to 
detain  it  a waiting  further  instructions. 

The  Soviets  immediately  responded  by  statinq  they  would 
absolutely  deny  any  boarqinq  of  the  freighter  and  any  such 
action  would  be  met  by  military  retaliation.  Intelligence 
has  also  revealed  the  ordering  of  at  least  ten  Soviet 
submarines  into  the  area  and  the  rapid  augmentation  of  its 
At  1  ant i c  Force  . 

The  ^resident  has  decided  he  can  not  allow  nuclear 
suoolies  to  reach  Cuba  and  he  feels  a  blockade  of  some  t.yoe 
must  be  d I  deed  around  Cuba.  He  immediately  Dlaced  the 
Atlantic  Fleet  on  alert  and  ordered  them  to  beqin  steaming 
towards  Cuba.  In  addition/  he  also  Placed  the  Mediterranean 


Fleet  on  alert 


T  ne  type 

and  size  of 

blockade  has 

not 

vet 

been 

determi  n  e  d . 

Ire  President 

has  asked 

you 

for 

your 

recommendation.  The  President  specifically  wants  a 
recommendation  as  to  whether  a  total  blockade  of  all 
shiooinq,  a  modified  blockade  of  some  sort  a  1  1  o  w  i  n  q  merchant 
ships  carryina  non-m i 1 i t a r y  caroo  to  pass,  or  a  roving 
oatrol  which  would  only  detain  ships  carrying  military 
hardware. 

0 Derations  personnel  have  stated  it  would  require  the 
entire  Atlantic  Fleet  ann  half  of  the  Mediterranean  Fleet  to 
invoke  a  total  blockade.  A  modified  blockade  would  reauire 
only  the  Atlantic  Fleet  while  the  rovina  cat  rol  would  only 
need  half  of  the  Atlantic  Fleet  resources. 

Intel  1 iaence  analysts  are  unsure  what  the  Soviet 
intentions  will  be  concernina  the  situation.  They  believe 
their  ootions  to  be  one  of  three! 

1.  An  attack  of  the  blockadina  forces. 

?.  An  attempt  to  run  through  the  blockade  without 
initiation  any  aodression»  or 

3.  Submission  to  the  blockade.  This  means  the 
Soviets  may  relent  to  having  some  shios  searched 
but  mav  have  others  return  directly  home. 

They  aoree,  however,  that  two  uocomina  events  will 
helc  to  determine  their  motives.  The  imoendina  vote  on 


t  ne  SALT  T  r ea  t  v  and  final  determination  of  exactly 
wnat  t  yce  of  co^strucHon  is  beinq  oone  on  the  coast  are  the 
kev  issues  in  as  session  their  intentions.  In  addition, 
tney  feel  SALT  Treaty  vote  will  he  effected  hv  noth  US 

oublic  ooinion  concerning  the  Treatv  and  current  world 
ooinion  concernino  the  situation.  The  UN  Security  council 
is  scheduled  to  meet  in  two  davs  to  discuss  the  situation. 

Toe  analysts  have  assessed  the  probability  of  US  Dublic 
ooinion  oeino  oro-SALT  as  70%  and  anti-SALT  as  beinq  30%. 
,'iorld  ooinion  is  assessed  as  having  a  40%  chance  of  being 
oro-US,  30%  oro-Soviet,  and  30%  of  oeino  neutral.  The 
analysts  also  currently  aaree  to  a  60%  likelihood  of  the 
construction  beinq  to  suooorf  full  nuclear  resuDDly,  30%  for 
only  Naval  r^sunoly  ( non - a r mamen t s  )  »  and  10%  as  a  new 
cohere  i  a  1  oort  . 

The  items  the  President  wants  considered  in  the 
selection  of  a  best  course  of  action  are  (in  order  of 
i moort  ance 1 : 

1)  National  security.  The  oresident  feels  he  can 

not  allow  the  Soviets  to  establish  a  suooly  base 
i n  Cuba. 

2 )  Safety  of  US  Nationals  currently  in  Cuba.  The 

US  has  recently  beaun  expo.tinq  numerous 

commercial  oroducts  to  Cuba  in  an  effort  to 
ooost  a  sagoinq  economy.  At  Dresent,  it  is 
estimated  there  are  at  least  2000  Americans  in 


Cuba 


US  oublic  opinion.  This  is  an  election  year  and 
analysts  feel  there  will  be  ou blic  outcry  it  we 
force  the  Soviets  into  a  military  confrontation# 
but  the  public  will  be  impressed  by  strong 
measures  which  force  the  Soviets  into  accepting 
our  demands. 

Cuban  reaction.  The  US  has  only  recently  begun 
trade  with  Cuba  and  is  anxious  to  continue  this 
if  at  all  possible.  Any  tyoe  of  blockade 
imposed  around  Cuba  will  have  a  serious  imoact 
on  these  trade  agreements. 

florid  opinion.  Any  confrontation  between  the  US 
and  USS^  is  undesirable.  If  the  US  is 
successful  with  the  blockade#  world  opinion  will 
be  high.  However#  if  we  provoke  aaqression  by 
our  actions#  we  will  lose  additional  world 
support  which  is  desperately  being  sought  by  the 


A  Ik  RECONNAISSANCE  SCENAPIO 


On  S  'Nr  c  n  1  P  8  0  ,  an  unarmed  Mohawk  reconnaissance 
aircraft  ahorten  its  mission  over  Korea.  It  returned  to  its 
base  where  the  crew  reoorted  an  interceotion  oy  a 
f i gh t e r - t yoe  aircraft  iron  North  Korea.  The  crew  of  the 
unarmed  Mohawk  had  not  stayed  around  long  enough  to 

determine  whether  the  interceoter  was  there  to  harass* 
snoot  >  identify,  or  had  just  "haooened  along"  while  on  a 

routine  training  fliaht.  Another  reconnaissance  flight  is 
scheduled  for  tomorrow  and  the  commander  needs  a 

recommendat i on  concerning  future  flights. 

After  further  discussion,  the  commander  desires  the 
f o 1 1 ow i no  notions  evaluated  in  light  of  what  has  raoDened 
and  what  the  intelligence  staff  conclude  are  North  Korea’s 
intentions: 

t.  Do  not  f 1 v  into  the  area  anymore. 

2 .  Fly  into  the  area  on  a  modified  route.  This 

modified  route  would  be  less  sensitive  but  also 
less  oroduct i ve  in  collection  ootential. 

3.  Fly  into  the  same  area  but  now  with  an  armed 
escort  . 

u.  Fly  into  the  area  with  h i oh-oer f ormance 
aircraft.  The  aircraft  would  be  less  vulnerable 
but  would  not  collect  data  as  well  as  the  normal 


ai rcraf t 


Fly  a  normal 


mission;  do  not  c^anae  anything 


5. 

from  previous  flights. 

jho  commander  is  mostly  concerned  with  the  Dossiole 
loss  of  the  aircraft  and  crew  durina  the  mission.  However* 
he  is  also  concerned  with  the  loss  of  prestige  or  political 
influence  if  flights  were  stooped  or  altered.  The  loss  of 
information  ana  the  oossible  imoact  n  the  reconnaissance 
orggram  in  other  areas  are  also  imoortant  factors. 

The  intelligence  section  believes  it  has  assessed  the 
Dossible  intentions  of  the  North  Korean  Government  on  future 


flights. 

Based  on  oast  situations  ana 

current 

c  apab i 1 i t i es  * 

they 

feel 

there  is  on  1 

1  v  a  slim 

chance 

(102)  the 

Koreans  will 

try 

to 

shoot  future 

a i rcraf t 

down,  a 

moderate 

chance  (30%) 

they 

will 

harass*  but 

orobab 1 y 

will  do 

nothing 

(60%)  . 

\ 


too 


APPFUQTx  C  -  SAMPLE  QUEST  I OMN  A  I  RES 


QUEST  1 0  -IN  a  I  wf  Fi'iw  TEA'''  vE-'-PF.PS  -.ITHO'JT  OPINT 
na*-E: 
r auk  : 

TEA'-'  *  : 

SCENARIO: 

xinAT  -ft  A  S  YOUR  RECO^Mf  NDE  0  ACTION? 


u y  Din  YOU  RECO-MVE;!i'  THAT  ACTION? 

PLEASE  ANSWER  THE  FOLLOWING  QUESTIONS  AS  INDICATED. 

1.  Do  vou  feel  this  scenario  was  realistic?  If  not  nl ease 
consent  in  the  soace  orovided. 

a .  Yes 

n.  No 


2.  /:ore  you  arte  to  identify  any  non-essent  i  a  1  information 
durinn  your  evaluation  of  the  courses  of  action?  If  yes» 
what  was  that  information? 

a .  YoS 

U .  No 


S.  Were  you  able  to  assess  the  criticality  of  the  factors 
which  contributed  to  the  choice  of  the  recommended  course 
of  action?  If  yes,  which  factors  were  the  most  critical? 

a .  Yes 

b .  No 


H.  Do  you  feel  confident  that  your  recommendation  was  the 
"best",  based  on  the  information  provided?  If  nr,  whv  not 

a .  Yes 


QuESTlUNNA  F  OR  T  t  A  J  E  '*1 d  F  P  S  "i  I  T  H  OP  INT 
name  : 

: 

T  E  A  v  a  ; 

SCENARIO: 

what  M A 3  YOUR  Rfc'COMMEMOEO  ACTION? 


fi  H  v  DIO  YOU  PECPVMEND  To  A  T  ACTION? 

PLEASE  ANSWER  The  FOLLOWING  QUESTIONS  AS  INDICATED. 

1.  Do  vou  ^eel  this  scenario  was  realistic?  If  no*  please 
comment  in  the  snace  orovi  Jeri. 

a .  Yes 

n .  u  o 


2.  ’Were  you  aole  to  identify  any  non-essential  information 
juring  your  evaluation  of  the  courses  of  action?  If  yes* 
what  was  t^at  i n f o r ^ t i on ? 

a .  Yps 

h .  'No 


3.  were  you  abl“  to  assess  the  criticality  of  the  factors 
which  contributed  to  the  choice  o  f  the  recommended  course 
of  action?  If  ves*  which  factors  were  the  most  critical? 

a .  Yes 


o 


M.  Do  vou  feel  confident  that  your  recommendation  was  the 
"best"*  based  on  the  information  provided?  If  no*  why  not? 


S.  Did  OP  [NT  oresent  the  evaluated  data  in  a  manner  which 
was  easily  un de r s t ood ?  It  no,  which  displays  were  difficult 
to  understand? 


a .  Yes 
o .  o 


6.  Did  OPIWT  display  its  results  a  u  i  c  k  1  y  enouqn?  If  not, 
which  displays  were  slow? 

a .  Yes 

b.  '-:o 

7.  "as  it  easy  to  mate  corrections  in  the  model?  If  not, 
comment  on  your  problems. 

a .  Yes 

0  .  'V  o 


8.  Did  QPI'JT  recover  well  from  any  errors  you  micjht  have 
made?  If  not i  what  was  the  result? 

a .  Yes 

b .  Do 

c.  D/i.  -  I  did  not  make  any  errors. 


103 


QUESTIONNAIRE  FUR  TEA'-'  '--EMBERS  AFTFP  COMPLETING  BOTH  SCENARIOS 


N  a  v  E  : 

RANK  ; 

T  F  A  w  a  ; 

PLEASE  ANSWER  I  HE  FOLLOWING  QUEST  IO.NS  AS  INDICATED. 

1.  To  what  extant  din  OP  I  NT  aid  in  your  assessment  of 
what  event  was  a  o  i  n  g  to  h  a  o  o  e  n  ? 

a.  Confused 

r.  .  \ o  Hein 

c .  Reinforced  own  ideas 
-i .  C  1  a  r  i  f  i  e  o 

e.  Enliohtenen 

f.  Had  no  need  to  assess  an  event 

2.  To  what  extent  did  OP  I  NT  aid  in  your  decision  process 
for  selectina  a  course  of  action? 

a.  Ignored 

n .  deli ed  uron  i t 

c.  Added  confidence 

d.  das  confusino 

3.  Do  you  thin*  tnat  OPINT  helped  to  speed  up  the  decision 
making  orocess? 

a .  Yes 

D  .  No 


a.  Do  you  think  you  would  rave  performed  Better  with  or 
without  OPINT? 

a.  :M i  t  h 

0  .  r.  i  t  h  o  u  t 


1  04 


5.  Do  vou  feel  that  OpIM’  coul  1  be  imnrcved  uoon?  If  yes» 
in  which  a  r  ®  a  s  ? 


a .  fes 

r,  .  '■!  o 

6.  Do  you  feel  decision  matters  should  use  CPINT.  or  should 
it  ce  left  to  technical  e*oerts? 

a.  Decision  makers 

b.  Technical  exoerts 

7  .  'hat  no  vou  oenceive  as  o e i no  the  areatest  advantaae  in 
u  s i no  DPI  ‘ J  T  ? 


8.  Anet  do  you  nerceive  as  heino  the  areatest  disadvantaoe 
in  u  s  i  n  a  0  0 1  'J  T  ? 


9.  a o u 1  a  you  use  QpIDT  in  the  future  if  it  was  made 
available  to  vou?  Please  comment. 
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